Develop and Deploy Mesos Hook

2020-01-23 00:00:00 +0000

Introduction

Background

We know that mesos agent is responsible for launching tasks, is it possible to do some checks before launching tasks without re-compiling mesos code ?

Yes! Developing shared libraries out of Mesos modules.

Mesos modules provide a way to easily extend inner workings of Mesos by creating and using shared libraries that are loaded on demand.

Mesos Modules

Here is the detail introduction regarding mesos module http://mesos.apache.org/documentation/latest/modules/, it supports a vast variety of modules, this article aims to introduce Hook.

Developing Hook

Usecase

User is able to write hooks for fulfilling the following functionalities:

  • docker mount path checking eg: for an illegal mount path hook will reject the task running
  • docker privileged checking eg: hook will check the flag and perform corresponding actions based on certain circumstances
  • etc.

Scaffold Codes

Below are the example hook codes, user can write their own logic in slavePreLaunchDockerTaskExecutorDecorator method:

#include <limits.h>
#include <stdlib.h>
#include <mesos/hook.hpp>
#include <mesos/mesos.hpp>
#include <mesos/module.hpp>

#include <mesos/module/hook.hpp>

#include <process/future.hpp>
#include <process/id.hpp>
#include <process/process.hpp>
#include <process/protobuf.hpp>

#include <stout/foreach.hpp>
#include <stout/os.hpp>
#include <stout/try.hpp>

#include "messages/messages.hpp"

#include <string>
#include <iostream>
#include <vector>

using namespace mesos;
using std::map;
using std::string;

using process::Failure;
using process::Future;

class MesosHook : public Hook {
public:
    MesosHook();

    /**
     * @param taskInfo
     * @param executorInfo
     * @param containerName
     * @param containerWorkDirectory
     * @param mappedSandboxDirectory
     * @param env
     * @return
     */
    virtual Future<Option<DockerTaskExecutorPrepareInfo>>
    slavePreLaunchDockerTaskExecutorDecorator(
            const Option<TaskInfo> &taskInfo,
            const ExecutorInfo &executorInfo,
            const string &containerName,
            const string &containerWorkDirectory,
            const string &mappedSandboxDirectory,
            const Option<map<string, string>> &env) {
        LOG(INFO) << "containerName: " + containerName;

        const Resource& resource = executorInfo.resources(0);
        LOG(INFO) << "resource role: " +resource.allocation_info().role();

        if (!taskInfo->has_container()) {
            return None();
        }

        if (taskInfo->container().type() == ContainerInfo::DOCKER) {
            LOG(INFO) << "Got a Docker Container";
        } else if (taskInfo->container().type() == ContainerInfo::MESOS) {
            LOG(INFO) << "Got a Mesos Container";
            // For now we ingore mesos container
            return None();
        } else {
            LOG(WARNING) << "Got a Unknown Type Container";
            return None();
        }

        if (!taskInfo->container().has_docker()) {
            return Failure("Check Docker Container failed");
        }
        ....
};

static Hook *createHook(const Parameters &parameters) {
    return new MesosHook();
}

mesos::modules::Module<Hook> xxx_Mesoshook(
        MESOS_MODULE_API_VERSION,
        MESOS_VERSION,
        "Company",
        "mail",
        "Despription",
        nullptr,
        createHook);

Compile Mesos Hook

Previously we installed the mesos environment successfully https://bluezd.github.io/archives/782, Now let’s head to compile mesos hook by taking advantage of existing mesos environment.

Replace variable MESOS_SOURCE_PATH and MESOS_INSTALL_PATH accordingly then execute the following shell script:

#!/bin/bash

set -e

MESOS_SOURCE_PATH="/root/mesos-1.9.0"
MESOS_BUILD_PATH="${MESOS_SOURCE_PATH}/build"
MESOS_INSTALL_PATH="/usr/local/mesos"

function compile_gcc () {
    if [[ -d build-gcc ]]; then
        rm -f  build-gcc/*.so
    else
        mkdir build-gcc
    fi

    cd build-gcc

    c++ -std=c++11 \
        -I $MESOS_INSTALL_PATH/lib/mesos/3rdparty/include \
        -I $MESOS_INSTALL_PATH/include/ \
        -I $MESOS_BUILD_PATH/src \
        -I $MESOS_SOURCE_PATH/src \
        -lmesos \
        -fpic \
        -o mesos_hook.o \
        -c ../mesos_hook.cpp

    gcc -shared -o libmesos_hook.so mesos_hook.o -lboost_filesystem -lboost_system
}

compile_gcc

The shared lib libmesos_hook.so will be placed in build-gcc dir.

Deploy Mesos Hook

prepare hook configuration file

Add following configurations to /etc/mesos/agent-modules.json:

     {
       "file": "/root/mesos-hook/build-gcc/libmesos_hook.so",
       "modules": [
         {
           "name": "xxx_Mesoshook"
         }
       ]
     }

The name must exactly same with mesos::modules::Module<Hook> xxx_Mesoshook in cpp file.

If you hope to pass parameters to your hook, add parameters defination as below:

     {
       "file": "/root/mesos-hook/build-gcc/libmesos_hook.so",
       "modules": [
         {
           "name": "xxx_Mesoshook",
           "parameters" :
              [
                {
                  "key": "xx",
                  "value" : "xxx"
                }
              ]
         }
       ]
     }

This the final file looks like:

# cat /etc/mesos/agent-modules.json
{
  "libraries":
  [
    {
      "file": "/usr/local/dcos-mesos-modules/lib/mesos/libmesos_network_overlay.so",
      "modules":
        [
          {
            "name": "com_mesosphere_mesos_OverlayAgentManager",
            "parameters" :
              [
                {
                  "key": "agent_config",
                  "value" : "/etc/mesos/overlay/agent-config.json"
                }
              ]
          }
        ]
     },
     {
       "file": "/root/mesos-hook/build-gcc/libmesos_hook.so",
       "modules": [
         {
           "name": "xxx_Mesoshook",
            "parameters" :
              [
                {
                  "key": "xx",
                  "value" : "xxx"
                }
              ]
         }
       ]
     }
  ]
}

starting mesos agent:

Add --hooks=xxx_Mesoshook to mesos agent startup options and start the agent:

#!/bin/bash

NODE_1_IP=172.16.9.101
NODE_2_IP=172.16.9.59

export PATH=$PATH:/usr/local/mesos/sbin/

mesos-agent \
    --containerizers=mesos,docker \
    --hostname=$NODE_2_IP \
    --image_providers=docker \
    --ip=0.0.0.0 \
    --isolation=docker/runtime,network/cni,filesystem/linux,volume/sandbox_path \
    --network_cni_config_dir=/etc/mesos/cni \
    --network_cni_plugins_dir=/etc/mesos/active/cni \
    --launcher_dir=/usr/local/mesos/libexec/mesos \
    --log_dir=/var/log/mesos \
    --master=zk://$NODE_1_IP:2181/mesos \
    --port=5051 \
    --work_dir=/var/lib/mesos/agent \
    --no-systemd_enable_support \
    --modules=file:///etc/mesos/agent-modules.json \
    --hooks=xxx_Mesoshook

What happend behind the scenes ?

mesos agent loads modules and initialize on startup

  1. Load modules Try<Nothing> result = ModuleManager::load(flags.modules.get()); in src/slave/main.cpp
    1. Try<Nothing> ModuleManager::loadManifest(const Modules& modules) in src/module/manager.cpp
    2. moduleBases[moduleName] = moduleBase;
  2. Initialize hooks Try<Nothing> result = HookManager::initialize(flags.hooks.get()); in src/slave/main.cpp
    1. Try<Hook*> module = ModuleManager::create<Hook>(hook); in src/hook/manager.cpp
      1. Module<T>* module = (Module<T>*) moduleBases[moduleName]; in src/module/manager.hpp
      2. T* instance = module->create(params.isSome() ? params.get() : moduleParameters[moduleName]); invoke createHook function
    2. availableHooks[hook] = module.get(); in src/hook/manager.cpp

call mesos hook logic on task startup

  • Future<Containerizer::LaunchResult> DockerContainerizerProcess::launch() in src/slave/containerizer/docker.cpp
    • f = HookManager::slavePreLaunchDockerTaskExecutorDecorator()
      • hook->slavePreLaunchDockerTaskExecutorDecorator() in src/hook/manager.cpp call user logic

The end.

Deploy mesos environment with overlay net from scratch

2020-01-21 00:00:00 +0000

Introduction

DC/OS is the perfect platform for running both docker and mesos workloads, it builds on top of apache mesos and supports additional advanced features (eg: network, service catalog, etc.) It provides comprehensive installers for both Cloud(aws, azure and google) and On-Prem. If users are interested in the principle under the hood they can also provision their own mesos env from scratch with advanced overlay network support(dcos-net powered by D2IQ).

This article aims to give you a guidence of deploying a pure mesos developing/testing environment with the following capabilities:

  1. overlay network by taking advantage of dcos-net module
  2. DNS resolution
    1. mesos-dns by taking advantage of mesos-dns module
    2. dcos-dns by taking advantage of dcos-dns(in dcos-net) module
  3. container orchestration platform by using marathon

Now Let’s get started.

Environment preparation

At least two machines:

  1. CentOS 7.6 (for mesos master) ip 172.16.9.101
  2. CentOS 7.6 (for mesos agent) ip 172.16.9.59

Install Depedencies on both master and slave

Execute the following commands:

#!/bin/bash

yum groupinstall -y "Development Tools"
yum install -y maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel \
               zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 \
               apr-devel subversion-devel apr-util-devel vim wget

yum install -y gcc-c++ \
               ncurses-devel libevent-devel systemd-devel libsodium-devel

yum install epel-release -y
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io

Compile and Install dcos-net on both master and slave

Install kerl

  1. curl -o /usr/local/bin/kerl https://raw.githubusercontent.com/kerl/kerl/master/kerl
  2. chmod a+x /usr/local/bin/kerl

Compiling Erlang

KERL_CONFIGURE_OPTIONS=
KERL_CONFIGURE_OPTIONS+="--enable-dirty-schedulers "
KERL_CONFIGURE_OPTIONS+="--enable-kernel-poll "
KERL_CONFIGURE_OPTIONS+="--with-ssl"
KERL_CONFIGURE_OPTIONS+="--without-javac "
export KERL_CONFIGURE_OPTIONS

kerl build 22.0 22.0
kerl install 22.0 $HOME/erl

# kerl status
Available builds:
22.0,22.0
----------
Available installations:
22.0 /root/erl
----------
The current active installation is:
/root/erl
There is no Dialyzer PLT for the active installation
The build options for the active installation are:

--enable-dirty-schedulers --enable-kernel-poll --with-ssl--without-javac

Compile and Install Mesos on both master and slave

  1. wget http://archive.apache.org/dist/mesos/1.9.0/mesos-1.9.0.tar.gz
  2. tar -zxvf mesos-1.9.0.tar.gz
  3. cd mesos-1.9.0
  4. mkdir build; cd build
  5. ../configure --enable-libevent --enable-ssl --enable-install-module-dependencies --enable-launcher-sealing --disable-libtool-wrappers --prefix=/usr/local/mesos/
  6. make -j $(getconf _NPROCESSORS_ONLN)
  7. make install

Compile and Install dc/os net overlay module on both master and slave

  1. yum install libsodium-devel -y
  2. git clone https://github.com/dcos/dcos-mesos-modules.git
  3. ./bootstrap
  4. mkdir build && cd build
  5. ../configure --with-mesos=/usr/local/mesos --prefix=/usr/local/dcos-mesos-modules --with-protobuf=/usr/local/mesos/lib/mesos/3rdparty
  6. make -j $(getconf _NPROCESSORS_ONLN)
  7. make install

Running a batch of services on master

starting exhibitor with zookeeper

#!/bin/bash

NODE_1_IP=172.16.9.101

docker run --rm -it -d \
     --name exhibitor \
     --net bridge \
     -p 2181:2181/tcp \
     -p 2888:2888/tcp \
     -p 3888:3888/tcp \
     -p 8181:8080/tcp \
     netflixoss/exhibitor:1.5.2 \
     --hostname $NODE_1_IP

starting mesos master

mkdir -p /etc/mesos/cni

#!/bin/bash

NODE_1_IP=172.16.9.101
export MESOS_MASTER=zk://localhost:2181/mesos

export PATH=$PATH:/usr/local/mesos/sbin/
mesos-master \
     --cluster=dcos-net-cluster \
     --hostname=$NODE_1_IP \
     --ip=0.0.0.0 \
     --log_dir=/var/log/mesos \
     --port=5050 \
     --quorum=1 \
     --zk=zk://$NODE_1_IP:2181/mesos \
     --work_dir=/var/lib/mesos/master \
     --modules=file:///etc/mesos/master-modules.json

# cat /etc/mesos/master-modules.json
{
  "libraries":
  [
    {
      "file": "/usr/local/dcos-mesos-modules/lib/mesos/libmesos_network_overlay.so",
      "modules":
        [
          {
            "name": "com_mesosphere_mesos_OverlayMasterManager",
            "parameters" :
              [
                {
                  "key": "master_config",
                  "value" : "/etc/mesos/overlay/master-config.json"
                }
              ]
          },
          {
            "name": "com_mesosphere_mesos_OverlayAgentManager",
            "parameters" :
              [
                {
                  "key": "agent_config",
                  "value" : "/etc/mesos/overlay/master-agent-config.json"
                }
              ]
          }
        ]
     }
  ]
}

# cat /etc/mesos/overlay/master-config.json
{
    "replicated_log_dir":"/var/lib/mesos/master",
    "network": {
        "vtep_mac_oui": "70:B3:D5:00:00:00",
        "overlays": [{
            "prefix": 24,
            "name": "dcos",
            "subnet": "9.0.0.0/8"
        }],
        "vtep_subnet": "44.128.0.0/20"
    }
}

# cat /etc/mesos/overlay/master-agent-config.json
{
    "cni_dir": "/etc/mesos/cni",
    "cni_data_dir": "/var/run/mesos/cni/networks",
    "network_config" : {
        "allocate_subnet": true,
        "mesos_bridge": false,
        "docker_bridge": false,
        "overlay_mtu": 1420,
        "enable_ipv6": false
    },
    "max_configuration_attempts": 5
}

starting mesos dns

  1. downlaod mesos dns from https://github.com/mesosphere/mesos-dns/releases
  2. /root/mesos-dns-v0.7.0-rc2-linux-amd64 -config /etc/mesos-dns.json
# cat /etc/mesos-dns.json
{
    "zk": "zk://172.16.9.101:2181/mesos",
    "refreshSeconds": 30,
    "ttl": 60,
    "domain": "mesos",
    "port": 61053,
    "resolvers": ["8.8.8.8"],
    "timeout": 5,
    "listener": "0.0.0.0",
    "email": "root.mesos-dns.mesos",
    "IPSources": ["host", "netinfo"],
    "SetTruncateBit": true
}

configuring and running dcos-net

Add the following entries to /etc/resolv.conf

nameserver 198.51.100.1
nameserver 198.51.100.2
nameserver 198.51.100.3

#!/bin/bash

modprobe ip_vs_wlc
modprobe dummy

iptables --wait -A FORWARD -j ACCEPT
iptables --wait -t nat -I POSTROUTING -m ipvs --ipvs --vdir ORIGINAL \
         --vmethod MASQ -m comment \
         --comment Minuteman-IPVS-IPTables-masquerade-rule \
         -j MASQUERADE
ip link add minuteman type dummy
ip link set minuteman up
ip link add spartan type dummy
ip link set spartan up
ip addr add 198.51.100.1/32 dev spartan
ip addr add 198.51.100.2/32 dev spartan
ip addr add 198.51.100.3/32 dev spartan

starting dcos-net

git clone https://github.com/dcos/dcos-net.git

#!/bin/bash

source $HOME/erl/activate

cd dcos-net
export MASTER_SOURCE=exhibitor
export EXHIBITOR_ADDRESS=172.16.9.101

./rebar3 shell \
        --config /etc/dcos-net/master.config \
        --name dcos-net@172.16.9.101 \
        --setcookie dcos-net \
        --apps dcos_dns,dcos_l4lb,dcos_overlay,dcos_rest,dcos_net

# cat /etc/dcos-net/master.config
[
{dcos_net,
  [{dist_port, 62501},
   {is_master, true}]},

 {dcos_dns,
  [{udp_port, 53},
   {tcp_port, 53},
   {zookeeper_servers, [{"172.16.9.101", 2181}]},
   {upstream_resolvers,
    [{{8,8,8,8}, 53}]},
   {zones,
    #{"component.thisdcos.directory" =>
          [#{
             type => cname,
             name => "registry",
             value => "master.dcos.thisdcos.directory"
            }]}}]},

 {dcos_l4lb,
  [{master_uri, "http://$172.16.9.101:5050/master/state"},
   {enable_lb, true},
   {enable_ipv6, false},
   {min_named_ip, {11,0,0,0}},
   {max_named_ip, {11,255,255,255}},
   {min_named_ip6, {16#fd01,16#c,16#0,16#0,16#0,16#0,16#0,16#0}},
   {max_named_ip6, {16#fd01,16#c,16#0,16#0,16#ffff,16#ffff,16#ffff,16#ffff}},
   {agent_polling_enabled, false}]},

 {dcos_overlay,
  [{enable_overlay, true},
   {enable_ipv6, false}]},

 {dcos_rest,
  [{enable_rest, true},
   {ip, {0,0,0,0}},
   {port, 62080}]},

 {erldns,
  [{servers, []},
   {pools, []}]},

 {telemetry,
  [{forward_metrics, false}]}
].

running marathon framework

download marathon 1.8

wget https://downloads.mesosphere.io/marathon/builds/1.8.222-86475ddac/marathon-1.8.222-86475ddac.tgz

run marathon

#!/bin/bash

export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/lib/libmesos.so
/root/marathon-1.8.222-86475ddac/bin/marathon --master zk://localhost:2181/mesos --zk=zk://localhost:2181/marathon

Running service on agent

running mesos agent

install container networking plugins:

  1. install golang env
    1. wget https://dl.google.com/go/go1.13.6.linux-amd64.tar.gz
    2. tar -C /usr/local -xzf go1.13.6.linux-amd64.tar.gz
    3. export PATH=$PATH:/usr/local/go/bin
  2. build container networking plugins
    1. git clone https://github.com/containernetworking/plugins.git
    2. cd plugins && ./build_linux.sh
  3. install container networking plugins
    1. mkdir -p /etc/mesos/active/cni/
    2. ln -s /root/plugins/bin/bridge /etc/mesos/active/cni/
    3. ln -s /root/plugins/bin/host-local /etc/mesos/active/cni/
    4. ln -s /usr/local/mesos/libexec/mesos/mesos-cni-port-mapper /etc/mesos/active/cni/

prepare conf file:

# cat /etc/mesos/agent-modules.json
{
  "libraries":
  [
    {
      "file": "/usr/local/dcos-mesos-modules/lib/mesos/libmesos_network_overlay.so",
      "modules":
        [
          {
            "name": "com_mesosphere_mesos_OverlayAgentManager",
            "parameters" :
              [
                {
                  "key": "agent_config",
                  "value" : "/etc/mesos/overlay/agent-config.json"
                }
              ]
          }
        ]
     }
  ]
}

# cat /etc/mesos/overlay/agent-config.json
{
    "master": "172.16.9.101:5050",
    "cni_dir":"/etc/mesos/cni",
    "cni_data_dir": "/var/run/mesos/cni/networks",
    "network_config": {
        "allocate_subnet": true,
        "mesos_bridge": true,
        "docker_bridge": false,
        "overlay_mtu": 1420,
        "enable_ipv6": false
    },
    "max_configuration_attempts": 5
}

# cat /etc/mesos/cni/dcos.conf
{
  "name": "dcos",
  "type": "mesos-cni-port-mapper",
  "excludeDevices": ["m-dcos"],
  "chain": "M-DCOS",
  "delegate": {
    "type": "bridge",
    "bridge": "m-dcos",
    "isGateway": true,
    "ipMasq": false,
    "hairpinMode": true,
    "mtu": 1420,
    "ipam": {
      "type": "host-local",
      "dataDir": "/var/run/mesos/cni/networks",
      "subnet": "9.0.0.0/25",
      "routes": [{
        "dst": "0.0.0.0/0"
      }]
    }
  }
}

starting mesos agent:

yum install ipset -y
systemctl enable docker;systemctl start docker
mkdir -p /var/run/mesos/cni/networks

#!/bin/bash

NODE_1_IP=172.16.9.101
NODE_2_IP=172.16.9.59

export PATH=$PATH:/usr/local/mesos/sbin/

mesos-agent \
    --containerizers=mesos,docker \
    --hostname=$NODE_2_IP \
    --image_providers=docker \
    --ip=0.0.0.0 \
    --isolation=docker/runtime,network/cni,filesystem/linux,volume/sandbox_path \
    --network_cni_config_dir=/etc/mesos/cni \
    --network_cni_plugins_dir=/etc/mesos/active/cni \
    --launcher_dir=/usr/local/mesos/libexec/mesos \
    --log_dir=/var/log/mesos \
    --master=zk://$NODE_1_IP:2181/mesos \
    --port=5051 \
    --work_dir=/var/lib/mesos/agent \
    --no-systemd_enable_support \
    --modules=file:///etc/mesos/agent-modules.json

configuring and running dcos-net

Add the following entries to /etc/resolv.conf

nameserver 198.51.100.1
nameserver 198.51.100.2
nameserver 198.51.100.3

#!/bin/bash

modprobe ip_vs_wlc
modprobe dummy

iptables --wait -A FORWARD -j ACCEPT
iptables --wait -t nat -I POSTROUTING -m ipvs --ipvs --vdir ORIGINAL \
         --vmethod MASQ -m comment \
         --comment Minuteman-IPVS-IPTables-masquerade-rule \
         -j MASQUERADE
ip link add minuteman type dummy
ip link set minuteman up
ip link add spartan type dummy
ip link set spartan up
ip addr add 198.51.100.1/32 dev spartan
ip addr add 198.51.100.2/32 dev spartan
ip addr add 198.51.100.3/32 dev spartan

starting dcos-net

git clone https://github.com/dcos/dcos-net.git

#!/bin/bash

source $HOME/erl/activate

cd dcos-net
export MASTER_SOURCE=exhibitor
export EXHIBITOR_ADDRESS=172.16.9.101

/root/dcos-net/rebar3 shell \
        --config /etc/dcos-net/agent.config \
        --name dcos-net@172.16.9.59 \
        --setcookie dcos-net \
        --apps dcos_dns,dcos_l4lb,dcos_overlay,dcos_rest,dcos_net

# cat /etc/dcos-net/agent.config
[
 {dcos_net,
  [{dist_port, 62501},
   {is_master, false}]},

 {dcos_dns,
  [{udp_port, 53},
   {tcp_port, 53},
   {mesos_resolvers, []},
   {zookeeper_servers, [{"172.16.9.101", 2181}]},
   {upstream_resolvers,
    [{{8,8,8,8}, 53}]}]},

 {dcos_l4lb,
  [{master_uri, "http://172.16.9.101:5050/master/state"},
   {enable_networking, true},
   {enable_lb, true},
   {enable_ipv6, false},
   {min_named_ip, {11,0,0,0}},
   {max_named_ip, {11,255,255,255}},
   {min_named_ip6, {16#fd01,16#c,16#0,16#0,16#0,16#0,16#0,16#0}},
   {max_named_ip6, {16#fd01,16#c,16#0,16#0,16#ffff,16#ffff,16#ffff,16#ffff}},
   {agent_polling_enabled, true}]},

 {dcos_overlay,
  [{enable_overlay, true},
   {enable_ipv6, false}]},

 {dcos_rest,
  [{enable_rest, true},
   {ip, {0,0,0,0}},
   {port, 62080}]},

 {erldns,
  [{servers, []},
   {pools, []}]}
].

Running tasks

accessing ui

  1. mesos ui http://master-ip:5050
  2. marathon ui http://master-ip:8080

running tasks in marathon

{
  "id": "/nginx-mesos",
  "cmd": null,
  "cpus": 1,
  "mem": 128,
  "disk": 0,
  "instances": 1,
  "container": {
    "type": "MESOS",
    "docker": {
      "forcePullImage": false,
      "image": "nginx:latest",
      "parameters": [],
      "privileged": false
    },
    "volumes": [],
    "portMappings": [
      {
        "containerPort": 0,
        "labels": {},
        "name": "default",
        "protocol": "tcp",
        "servicePort": 10000
      }
    ]
  },
  "networks": [
    {
      "name": "dcos",
      "mode": "container"
    }
  ],
  "portDefinitions": [],
  "maxLaunchDelaySeconds": 300
}

verifying results

# ping nginx-mesos.marathon.containerip.dcos.thisdcos.directory
PING nginx-mesos.marathon.containerip.dcos.thisdcos.directory (9.0.2.2) 56(84) bytes of data.
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=1 ttl=64 time=0.041 ms
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=3 ttl=64 time=0.041 ms
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=4 ttl=64 time=0.042 ms
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=5 ttl=64 time=0.043 ms
64 bytes from 9.0.2.2 (9.0.2.2): icmp_seq=6 ttl=64 time=0.040 ms

# curl nginx-mesos.marathon.containerip.dcos.thisdcos.directory
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

The end.

通过 Jekyll 迁移 Blog

2017-09-15 00:00:00 +0000

Why ?

为什么不用 wordpress 了呢？先说说痛点:

  1. wordpress 安装较为繁琐 (LAMP)
  2. 需要定期维护，比如备份数据库
  3. 迁移过程很繁琐，环境需要重新安装配置。

Jekyll 优势

jekyll 是一个基于 Ruby 开发的简单的免费的Blog生成工具, 可以通过它把 wordpress blog 转换成静态网页，不需要数据库。

可以通过如下方式进行 migrate:

  • 官方脚本
  • 或者在 wordpress 上安装插件 jekyll-exporter 导出成一个 jekyll-export.zip 的文件
  • 评论的话可以通过 Disqus 迁移, 像我这样没啥评论的就不用费这劲了。

好处有哪些呢？

  • 用 markdown 写 blog
  • 所有 posts 可以保存在 git respository 中, 发布新的文章很方便
  • 不需要定期维护备份

部署

部署在哪呢 ?

  • github pages
    可以免费部署在这里，通过 git 发布维护 blog, 很是方便
  • VPS
    当然也可以部署在 VPS上，不过你要事先安装好环境，比如 ruby, jekyll 等等。 那如果以后迁移到其他环境岂不是又要配置环境吗 ? 是的，不过我把 jekyll 运行环境做成了一个 Docker image, 通过如下方式运行你的 blog:
    • docker pull bluezd/jekyll-build
    • 把你的博客 git clone 到某个路径下 /Path/To
    • docker run --name blog -p 80:4000 -v /Path/To:/jekyll-home/blog bluezd/jekyll-build:latest

    这样你的 blog 就能跑起来了，是不是很简单，你不需要在配置环境上花费时间，把省下的时间放在写 blog 上。

SIGNAL_UNKILLABLE

2016-08-10 13:30:07 +0000

众所周知 SIGKILL 信号不能被捕获，可以通过如下方法给进程发送此信号来结束它:

kill -s SIGKILL `pgrep XXX`

那么问题来了，可以给 init 进程发送此信号结束它吗? 显然是不能，如果 init 进程挂了，系统不就挂了吗?

验证一下:

kill -s SIGKILL 1

init 进程也就是 systemd 还在那里，没有任何改变。

原因就是内核在创建 init 进程的时候设置了 SIGNAL_UNKILLABLE 这个 flag:

kernel/fork.c
--

static struct task_struct *copy_process()
{
        ...
        if ((clone_flags & CLONE_SIGHAND) && !(clone_flags & CLONE_VM))
                 return ERR_PTR(-EINVAL);
        ...
        if (likely(p->pid)) {
                ptrace_init_task(p, (clone_flags & CLONE_PTRACE) || trace);

                init_task_pid(p, PIDTYPE_PID, pid);
                if (thread_group_leader(p)) {
                        init_task_pid(p, PIDTYPE_PGID, task_pgrp(current));
                        init_task_pid(p, PIDTYPE_SID, task_session(current));

                        if (is_child_reaper(pid)) {
                                ns_of_pid(pid)->child_reaper = p;
                                p->signal->flags |= SIGNAL_UNKILLABLE;
                        }
        ...
}

内核 signal 相应处理:

kernel/signal.c
--

static void complete_signal(int sig, struct task_struct *p, int group)
{
        ...
        if (sig_fatal(p, sig) &&
            !(signal->flags & (SIGNAL_UNKILLABLE | SIGNAL_GROUP_EXIT)) &&
            !sigismember(&t->real_blocked, sig) &&
            (sig == SIGKILL || !t->ptrace)) {
                ...
        }
        ...
}

写了个内核模块用于测试，主要是检索所有进程，查看是否设置了 SIGNAL_UNKILLABLE flag, 如果设置了，打引出进程名字，并且如果 pid == 1 的话则 clear 这个 flag.

生成了模块后通过如下方法对模块进行签名(不签名也可以，取决于内核选项，一般也可以载入成功):

CONFIG_MODULE_SIG_HASH="sha256"

./sign-file sha256 ./signing_key.priv ./signing_key.x509 /path/to/XXX.ko

signing_key.priv 和 signing_key.x509是在编译内核的时候通过 x509.genkey 配置文件生成的，通过如下命令:

kernel/Makefile
--

openssl req -new -nodes -utf8 -$(CONFIG_MODULE_SIG_HASH) -days 36500 \
        -batch -x509 -config x509.genkey \
        -outform DER -out signing_key.x509 \
        -keyout signing_key.priv 2>&1

然后内核把 signing_key.x509(公钥) 写到 x509_certificate_list最后加载到内核中去，当手工载入模块的时候 内核会通过这个公钥对这个模块进行验证。所以在生成模块后需要用那两个公钥和私钥对模块进行签名，以便内核 可以验证并且知道这个模块是安全的。

Summary:

  1. 在 Bare Metal系统中只有 init 进程设置了 SIGNAL_UNKILLABLE flag.
  2. clear 这个 flag 后如果发送 SIGKILL 信号 init 进程, 进程被 kill 掉了，系统也 panic 挂掉了。

五一游

2015-05-06 20:43:54 +0000

好久没有写了，这次真的是好久没有写了，越来越懒了。

回到大连已经一年多了，每天步行上下班，住自己家，不用租房，很是舒服。可是生活啊不可能没有问题，总是一个接着一个，咋办呢？坦然面对就好。

年初败了一块手表，最近又败了一个相机，虽然都是我梦寐以求想要的，可是真心心疼啊。据说一个男人一生中拥有美酒、跑车、相机、金表才算完整，目前还差两样，等以后发家致富中彩票后再考虑吧。

五一前败的微单 Sony A7M2，看了又看等了又等，最终还是横下一条心，买！走上了这条不归路(第二天就看上了蔡司35的定焦头)。。。。。。 买完后各种研究，趁着五一和家人出去溜达赏花，顺便也实战演练下。

多说无益，上图！

想要拍出一张好的照片很难，尤其是对于一个摄影菜鸟来说，要学习的东西还很多！