Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GetNic Will return "not found" wrongly #32

Open
Smithx10 opened this issue Jun 26, 2019 · 8 comments
Open

GetNic Will return "not found" wrongly #32

Smithx10 opened this issue Jun 26, 2019 · 8 comments

Comments

@Smithx10
Copy link

While Extending https://github.com/terraform-providers/terraform-provider-triton to accept NetworkObjects I discovered some intermittent behavior here. I created the following test to demonstrate what was occuring.

The following test demonstrates:
A NIC is added to a newly created running Instance.
We begin polling for a state of "running"
We find the NIC that we added with the state of "provisioning" for awhile
The NIC is no longer found for awhile
We find the NIC that we added with the state of "running"

Test:

        inst, _ := c.Instances().Create(context.Background(), &compute.CreateInstanceInput{
                Name:    "Nic-test-5",
                Image:   "3dbbdcca-2eab-11e8-b925-23bf77789921",
                Package: "sample-2G",
        })

        for {
                running, _ := c.Instances().Get(context.Background(), &compute.GetInstanceInput{
                        ID: inst.ID,
                })

                if running.State == "running" {
                        break
                }

                time.Sleep(2 * time.Second)
        }

        nic, err := c.Instances().AddNIC(context.Background(), &compute.AddNICInput{
                InstanceID: inst.ID,
                NetworkObject: compute.NetworkObject{
                        IPv4UUID: "0a63f067-d721-435e-941a-724776d69c91",
                        //IPv4IPs:  []string{"10.1.1.58"},
                },
                //Network: "71a1abac-f003-4b51-ac63-28d37f2ef0af",
        })

        fmt.Println(nic, err)

        for {
                nicrunning, err := c.Instances().GetNIC(context.Background(), &compute.GetNICInput{
                        InstanceID: inst.ID,
                        MAC:        nic.MAC,
                })

                fmt.Println(nicrunning, err)

                if err == nil {
                        if nicrunning.State == "running" {
                                goto X
                                break
                        }
                }

                time.Sleep(1 * time.Second)
        }
X:
}

Result:

arch@archlinux ~/g/s/g/test ❯❯❯ go run main.go                                                                                        ✘ 1
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 provisioning 0a63f067-d721-435e-941a-724776d69c91} <nil>
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
<nil> unable to get machine NIC: ResourceNotFound: nic not found
&{10.1.1.91 90:b8:d0:81:68:21 false 255.255.255.0 10.1.1.1 running 0a63f067-d721-435e-941a-724776d69c91} <nil>
@Smithx10 Smithx10 mentioned this issue Jun 26, 2019
Open
@Smithx10 Smithx10 changed the title GetNic Will return not found wrongly GetNic Will return "not found" wrongly Jun 26, 2019
@twhiteman
Copy link
Contributor

@Smithx10 what type of image/instance is this (e.g. Ubuntu distro, LX, SmartOS, KVM)?

@Smithx10
Copy link
Author

https://docs.joyent.com/public-cloud/instances/infrastructure/images/centos#centos-7-20180323

LX, Centos-7

@twhiteman
Copy link
Contributor

twhiteman commented Jul 2, 2019 •
edited
Loading

@Smithx10 I'm having trouble reproducing this (I don't get the "unable to get machine NIC: ResourceNotFound: nic not found" error, I just get nic states "provisioning" and "running").

What nics are on the machine after the instance.Create step? I.e. running.Networks when the instance has become running.

@Smithx10
Copy link
Author

Smithx10 commented Jul 3, 2019 •
edited
Loading

@twhiteman @mgerdts, I am going to test this with another machine image. Currently the workflow job for addinging the NIC is running into the "vmadam shutdown not exiting 0" bug that @mgerdts addressed earlier in the week. Perhaps that failure in the workflow is causing this condition? Todd are you having any errors in your workflow job?

The following task timed out after 90s.
vmapi.check_updated
task timeout error

HTTP/1.1 200 OK
request-id: 34c837b1-5502-46d2-9ced-20a10c88b325
Content-Type: application/json
Content-Length: 52193
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, Api-Version, Response-Time
Access-Control-Allow-Methods: GET, HEAD
Access-Control-Expose-Headers: Api-Version, Request-Id, Response-Time
Connection: Keep-Alive
Content-MD5: 3xMSUWwmJSF98c6inGvnXQ==
Date: Wed, 03 Jul 2019 13:26:16 GMT
Server: WorkflowAPI
Response-Time: 69
x-request-id: e9b00eb2-bdce-4b6f-ae3c-a890bf4e3d5d
x-response-time: 70
x-server-name: ae194cba-84ad-4781-bab3-46b316c0f305

{
  "execution": "failed",
  "chain_results": [
    {
      "result": "All parameters OK!",
      "error": "",
      "name": "common.validate_params",
      "started_at": "2019-07-03T13:19:17.570Z",
      "finished_at": "2019-07-03T13:19:17.602Z"
    },
    {
      "result": "Request has been setup!",
      "error": "",
      "name": "common.setup_request",
      "started_at": "2019-07-03T13:19:17.729Z",
      "finished_at": "2019-07-03T13:19:17.761Z"
    },
    {
      "result": "OK",
      "error": "",
      "name": "napi.validate_networks",
      "started_at": "2019-07-03T13:19:17.909Z",
      "finished_at": "2019-07-03T13:19:17.940Z"
    },
    {
      "result": "OK",
      "error": "",
      "name": "cnapi.get_server_nic_tags",
      "started_at": "2019-07-03T13:19:18.089Z",
      "finished_at": "2019-07-03T13:19:18.983Z"
    },
    {
      "result": "Server has all the required NIC tags",
      "error": "",
      "name": "napi.check_server_nic_tags",
      "started_at": "2019-07-03T13:19:19.139Z",
      "finished_at": "2019-07-03T13:19:19.361Z"
    },
    {
      "result": "NICs looked up or allocated",
      "error": "",
      "name": "napi.add_nics_by_mac",
      "started_at": "2019-07-03T13:19:19.479Z",
      "finished_at": "2019-07-03T13:19:19.701Z"
    },
    {
      "result": "AddNics called with macs -- skipping",
      "error": "",
      "name": "napi.provision_nics",
      "started_at": "2019-07-03T13:19:19.869Z",
      "finished_at": "2019-07-03T13:19:19.892Z"
    },
    {
      "result": "No fabric NICs",
      "error": "",
      "name": "cnapi.acquire_fabric_nat_tickets",
      "started_at": "2019-07-03T13:19:20.018Z",
      "finished_at": "2019-07-03T13:19:20.049Z"
    },
    {
      "result": "No fabric NATs to provision",
      "error": "",
      "name": "napi.provision_fabric_nats",
      "started_at": "2019-07-03T13:19:20.178Z",
      "finished_at": "2019-07-03T13:19:20.207Z"
    },
    {
      "result": "Added network parameters to payload",
      "error": "",
      "name": "common.update_network_params",
      "started_at": "2019-07-03T13:19:20.350Z",
      "finished_at": "2019-07-03T13:19:20.581Z"
    },
    {
      "result": "OK",
      "error": "",
      "name": "cnapi.acquire_vm_ticket",
      "started_at": "2019-07-03T13:19:20.699Z",
      "finished_at": "2019-07-03T13:19:21.060Z"
    },
    {
      "result": "OK",
      "error": "",
      "name": "cnapi.wait_on_vm_ticket",
      "started_at": "2019-07-03T13:19:21.208Z",
      "finished_at": "2019-07-03T13:19:21.238Z"
    },
    {
      "result": "No fabric NATs provisioned",
      "error": "",
      "name": "cnapi.wait_for_fabric_nat_provisions",
      "started_at": "2019-07-03T13:19:21.388Z",
      "finished_at": "2019-07-03T13:19:21.418Z"
    },
    {
      "result": "Task id: d94a66ee-2640-caae-b8c2-ab554f27807e queued to CNAPI!",
      "error": "",
      "name": "cnapi.update_vm",
      "started_at": "2019-07-03T13:19:21.558Z",
      "finished_at": "2019-07-03T13:19:21.879Z"
    },
    {
      "result": "Job succeeded!",
      "error": "",
      "name": "cnapi.wait_task",
      "started_at": "2019-07-03T13:19:22.028Z",
      "finished_at": "2019-07-03T13:19:51.059Z"
    },
    {
      "result": "",
      "error": "task timeout error",
      "name": "vmapi.check_updated",
      "started_at": "2019-07-03T13:19:51.225Z",
      "finished_at": "2019-07-03T13:21:21.250Z"
    }
  ],
  "params": {
    "macs": [
      "90:b8:d0:b4:20:bd"
    ],
    "task": "add_nics",
    "vm_uuid": "f0e54000-725a-4224-a0a3-c4d7138c4f7d",
    "owner_uuid": "2331453d-5310-42a1-dd63-92dcc4004f0d",
    "server_uuid": "38363837-3034-324d-3238-353130313433",
    "last_modified": "2019-07-03T13:17:20.000Z",
    "oldResolvers": [
      "10.45.137.14",
      "10.45.137.15"
    ],
    "wantResolvers": true,
    "x-request-id": "ebf3cf4d-2dc8-4381-a43a-3d98eeee6bf4",
    "sdc_nat_pool": "a99cbb20-3bf2-4469-8236-81862b0a9c7b",
    "filteredNetworks": {
      "netInfo": [],
      "networks": [],
      "fabrics": [],
      "pools": [],
      "nics": []
    },
    "context": {
      "caller": {
        "type": "signature",
        "ip": "::ffff:127.0.0.1",
        "keyId": "/bruce_dev/keys/4c:e6:33:6b:b3:c7:ef:c0:c2:0f:d9:f5:fc:ac:c3:a6"
      }
    },
    "creator_uuid": "2331453d-5310-42a1-dd63-92dcc4004f0d",
    "origin": "cloudapi",
    "jobid": "34c837b1-5502-46d2-9ced-20a10c88b325",
    "fabricNatNics": [],
    "add_nics": [
      {
        "belongs_to_type": "zone",
        "belongs_to_uuid": "f0e54000-725a-4224-a0a3-c4d7138c4f7d",
        "mac": "90:b8:d0:b4:20:bd",
        "owner_uuid": "2331453d-5310-42a1-dd63-92dcc4004f0d",
        "primary": false,
        "state": "provisioning",
        "created_timestamp": "2019-07-03T13:19:14.999Z",
        "modified_timestamp": "2019-07-03T13:19:14.999Z",
        "ip": "10.1.1.95",
        "fabric": true,
        "gateway": "10.1.1.1",
        "gateway_provisioned": true,
        "internet_nat": true,
        "mtu": 8500,
        "netmask": "255.255.255.0",
        "nic_tag": "sdc_overlay/13954390",
        "resolvers": [
          "10.45.137.14",
          "10.45.137.15"
        ],
        "vlan_id": 100,
        "routes": {},
        "network_uuid": "0a63f067-d721-435e-941a-724776d69c91",
        "cn_uuid": "38363837-3034-324d-3238-353130313433"
      }
    ],
    "resolvers": [
      "10.45.137.14",
      "10.45.137.15"
    ]
  },
  "task": "add_nics",
  "target": "/add-nics-f0e54000-725a-4224-a0a3-c4d7138c4f7d",
  "vm_uuid": "f0e54000-725a-4224-a0a3-c4d7138c4f7d",
  "server_uuid": "38363837-3034-324d-3238-353130313433",
  "creator_uuid": "2331453d-5310-42a1-dd63-92dcc4004f0d",
  "origin": "cloudapi",
  "workflow": "cfbae18e-059f-4f05-a566-50886365a5a5",
  "exec_after": null,
  "num_attempts": 0,
  "name": "add-nics-7.2.2",
  "version": "7.2.2",
  "chain": [
    {
      "name": "common.validate_params",
      "timeout": 10,
      "retry": 1,
      "body": "function validateForZoneAction(job, cb) {\n    if (!cnapiUrl) {\n        cb('No CNAPI URL provided');\n        return;\n    }\n\n    if (!job.params['vm_uuid']) {\n        cb('VM UUID is required');\n        return;\n    }\n\n    cb(null, 'All parameters OK!');\n}",
      "modules": {}
    },
    {
      "name": "common.setup_request",
      "timeout": 10,
      "retry": 1,
      "body": "function setupRequest(job, cb) {\n    job.endpoint = '/servers/' +\n                   job.params['server_uuid'] + '/vms/' +\n                   job.params['vm_uuid'] + '/update';\n    job.params.jobid = job.uuid;\n    job.requestMethod = 'post';\n    job.action = 'add_nics';\n    job.server_uuid = job.params['server_uuid'];\n\n    return cb(null, 'Request has been setup!');\n}",
      "modules": {}
    },
    {
      "name": "napi.validate_networks",
      "timeout": 10,
      "retry": 1,
      "body": "function validateNetworks(job, cb) {\n    var networks = job.params.networks;\n    var filteredNetworks = job.params.filteredNetworks;\n\n    // add-nics also calls this function, but if macs are provided we don't\n    // necessarily need to progress further\n    if (job.params.macs && !networks) {\n        return cb();\n    }\n\n    job.nicTagReqs = [];\n\n    function pushTags(net) {\n        if (net.nic_tags_present) {\n            job.nicTagReqs.push(net.nic_tags_present);\n        } else {\n            job.nicTagReqs.push([ net.nic_tag ]);\n        }\n    }\n\n    filteredNetworks.netInfo.forEach(function (net) {\n        pushTags(net);\n    });\n\n    job.log.info({ nicTagReqs: job.nicTagReqs },\n        'NIC Tag requirements retrieved');\n\n    cb(null, 'NIC Tag requirements retrieved');\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "cnapi.get_server_nic_tags",
      "timeout": 10,
      "retry": 1,
      "body": "function getServerNicTags(job, cb) {\n    function extractServerNicTags(err, server) {\n        if (err) {\n            cb(err);\n            return;\n        }\n\n        /*\n         * Perform the same logic that DAPI performs on the sysinfo payload,\n         * minus the parts about online/offline NICs, since we're either\n         * adding new NICs to a VM or performing a manual server selection.\n         */\n\n        var interfaces = server.sysinfo['Network Interfaces'] || {};\n        var vnics = server.sysinfo['Virtual Network Interfaces'] || {};\n\n        var serverTags = {};\n\n        Object.keys(interfaces).forEach(function extractTags(nicName) {\n            var nic = interfaces[nicName];\n            var nicTags = nic['NIC Names'];\n\n            for (var i = 0; i < nicTags.length; i++) {\n                serverTags[nicTags[i]] = true;\n            }\n        });\n\n        Object.keys(vnics).forEach(function extractOverlayTags(nicName) {\n            var nic = vnics[nicName];\n            var nicTags = nic['Overlay Nic Tags'] || [];\n\n            for (var i = 0; i < nicTags.length; i++) {\n                serverTags[nicTags[i]] = true;\n            }\n        });\n\n        job.serverNicTags = Object.keys(serverTags);\n\n        cb();\n    }\n\n    if (job.server_info) {\n        extractServerNicTags(null, job.server_info);\n    } else {\n        var cnapi = new sdcClients.CNAPI({\n            url: cnapiUrl,\n            headers: { 'x-request-id': job.params['x-request-id'] }\n        });\n\n        cnapi.getServer(job.params.server_uuid, extractServerNicTags);\n    }\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "napi.check_server_nic_tags",
      "timeout": 10,
      "retry": 1,
      "body": "function checkServerNicTags(job, cb) {\n    function done(err) {\n        if (err) {\n            cb(err);\n        } else {\n            cb(null, 'Server has all the required NIC tags');\n        }\n    }\n\n    var macs = job.params.macs;\n\n    if (macs) {\n        /*\n         * If 'macs' was passed, we're dealing with pre-created NICs, so we need\n         * to pull the NICs from NAPI first.\n         */\n        var napi = new sdcClients.NAPI({\n            url: napiUrl,\n            headers: { 'x-request-id': job.params['x-request-id'] }\n        });\n\n        async.mapSeries(macs, function lookupMAC(mac, next) {\n            napi.getNic(mac, function checkTagOkay(err, nic) {\n                if (err) {\n                    next(err);\n                    return;\n                }\n\n                var nicTag = nic.nic_tag;\n\n                if (!nicTag) {\n                    next(new Error('NIC ' + mac + 'does not have a tag'));\n                    return;\n                }\n\n                /*\n                 * This hack is to split the NIC tag off from the vnet_id, which\n                 * fabric NICs have embedded in their nic_tag attribute.\n                 */\n                var overlay = nicTag.match(/^(.+)\\/\\d+$/);\n                nicTag = overlay ? overlay[1] : nicTag;\n\n                if (job.serverNicTags.indexOf(nicTag) === -1) {\n                    next(new Error('Server does not have NIC tag: ' + nicTag));\n                    return;\n                }\n\n                next();\n            });\n        }, done);\n    } else {\n        /*\n         * Otherwise we're dealing with networks. The nic_tag requirements for\n         * these networks and pools were already loaded by validateNetworks().\n         * We need to make sure that the specified server satisfies at least one\n         * of the tags required for each network and pool.\n         */\n        var serverTags = {};\n        job.serverNicTags.forEach(function extractServerTag(tag) {\n            serverTags[tag] = true;\n        });\n\n        for (var i = 0; i < job.nicTagReqs.length; i++) {\n            var reqs = job.nicTagReqs[i];\n            var satisfied = false;\n\n            for (var j = 0; j < reqs.length; j++) {\n                if (serverTags[reqs[j]]) {\n                    satisfied = true;\n                    break;\n                }\n            }\n\n            if (!satisfied) {\n                done(new Error(\n                    'Server must have one of the following NIC tags: ' +\n                    reqs.join(', ')));\n                return;\n            }\n        }\n\n        done();\n        return;\n    }\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "napi.add_nics_by_mac",
      "timeout": 20,
      "retry": 1,
      "body": "function addNicsByMac(job, cb) {\n    var filteredNetworks = job.params.filteredNetworks;\n    var networks = filteredNetworks.networks;\n    var macs     = job.params.macs;\n\n    job.params.fabricNatNics = [];\n\n    // If this is a nic on a fabric, has no gateway provisioned, and the network\n    // requests an internet NAT, add it\n    function addFabricNatNic(fNic) {\n        if (fNic && fNic.fabric && fNic.gateway && !fNic.gateway_provisioned &&\n                fNic.ip !== fNic.gateway && fNic.internet_nat) {\n            job.params.fabricNatNics.push(fNic);\n        }\n    }\n\n    if (networks === undefined && macs === undefined) {\n        cb('Networks or mac are required');\n        return;\n    }\n\n    if (!macs) {\n        cb(null, 'AddNics not called with macs -- skipping');\n        return;\n    }\n\n    var napi = new sdcClients.NAPI({\n        url: napiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    var nics = [];\n\n    function done(err) {\n        if (err) {\n            cb(err);\n        } else {\n            job.log.info({ nics: nics }, 'NICs allocated');\n            job.params['add_nics'] = nics;\n\n            cb(null, 'NICs looked up or allocated');\n        }\n    }\n    async.mapSeries(macs, function (mac, next) {\n        napi.getNic(mac, function (err, nic) {\n            if (err) {\n                return next(err);\n            }\n\n            nics.push(nic);\n            addFabricNatNic(nic);\n            next();\n        });\n    }, done);\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "napi.provision_nics",
      "timeout": 20,
      "retry": 1,
      "body": "function provisionNics(job, cb) {\n    var macs = job.params.macs;\n\n    if (macs) {\n        cb(null, 'AddNics called with macs -- skipping');\n        return;\n    }\n    var inst_uuid = job.params.uuid || job.params.vm_uuid;\n    var owner_uuid = job.params.owner_uuid;\n\n    var filteredNetworks = job.params.filteredNetworks;\n    if (filteredNetworks === undefined) {\n        cb('Networks should be populated already');\n        return;\n    }\n\n    var networks = filteredNetworks.networks;\n    networks.forEach(function (net) {\n        // Make absolutely sure we're never overriding NAPI's network\n        // owner checks:\n        delete net.check_owner;\n    });\n\n    var poolNetworks = filteredNetworks.networks.filter(function\n        filterOutPoolNets(net) {\n        if (filteredNetworks.pools.indexOf(net.ipv4_uuid) >= 0) {\n            return true;\n        }\n        return false;\n    });\n\n    var napi = new sdcClients.NAPI({\n        url: napiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    // Every NIC we provision or updated is added to this array\n    var nics = filteredNetworks.nics.slice();\n    job.params.fabricNatNics = [];\n\n\n    // Return a new copy for every time we provision a new NIC and avoid\n    // accidentally reusing an object\n    function nicParams() {\n        return {\n            owner_uuid: owner_uuid,\n            belongs_to_uuid: inst_uuid,\n            belongs_to_type: 'zone',\n            state: 'provisioning',\n            nic_tags_available: job.serverNicTags,\n            cn_uuid: job.params.server_uuid\n        };\n    }\n\n    // If this is a nic on a fabric, has no gateway provisioned, and the network\n    // requests an internet NAT, add it\n    function addFabricNatNic(fNic) {\n        if (fNic && fNic.fabric && fNic.gateway && !fNic.gateway_provisioned &&\n                fNic.ip !== fNic.gateway && fNic.internet_nat) {\n            job.params.fabricNatNics.push(fNic);\n        }\n    }\n\n    var antiSpoofParams = [\n        'allow_dhcp_spoofing',\n        'allow_ip_spoofing',\n        'allow_mac_spoofing',\n        'allow_restricted_traffic'\n    ];\n\n    job.params.nics = nics;\n\n    /*\n     * We only provision NICs for network_pools below.  The NICs for networks\n     * are provisioned in createVm()->preProvisionNics().\n     */\n    async.series([\n        function provisionPoolNicss(callback) {\n            async.mapSeries(poolNetworks, function (network, next) {\n                var params = nicParams();\n                if (network.ipv4_ips !== undefined)\n                    params.ip = network.ipv4_ips[0];\n                if (network.primary !== undefined)\n                    params.primary = network.primary;\n\n                antiSpoofParams.forEach(function (spoofParam) {\n                    if (network.hasOwnProperty(spoofParam)) {\n                        params[spoofParam] = network[spoofParam];\n                    }\n                });\n\n                napi.provisionNic(network.ipv4_uuid, params,\n                    function (suberr, nic) {\n                    if (suberr) {\n                        next(suberr);\n                    } else {\n                        nics.push(nic);\n                        // pools may contain fabric networks someday\n                        addFabricNatNic(nic);\n                        next();\n                    }\n                });\n            }, callback);\n        },\n        function updateExistingNics(callback) {\n            // Update existing nics\n            async.mapSeries(filteredNetworks.nics, function (nic, next) {\n                var params = nicParams();\n\n                napi.updateNic(nic.mac, params,\n                    function (suberr, updatedNic) {\n                    if (suberr) {\n                        next(suberr);\n                    } else {\n                        addFabricNatNic(updatedNic);\n                        next();\n                    }\n                });\n            }, callback);\n        }\n    ], function (err, results) {\n        if (err) {\n            cb(err);\n        } else {\n            job.log.info({ nics: job.params.nics }, 'NICs allocated');\n\n            // If we hit this due to add_nics setup params so that we call\n            // common.update_network_params properly\n            if (job.task === 'add_nics') {\n                job.params['add_nics'] = nics;\n            }\n\n            cb(null, 'NICs allocated and updated');\n        }\n    });\n\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "cnapi.acquire_fabric_nat_tickets",
      "timeout": 10,
      "retry": 1,
      "body": "function acquireFabricTickets(job, cb) {\n    if (!job.params.fabricNatNics || job.params.fabricNatNics.length === 0) {\n        return cb(null, 'No fabric NICs');\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var nics = [];\n    var netuuids = [];\n\n    job.params.fabricNatTickets = [];\n\n    // Uniquify, just in case\n    for (var n in job.params.fabricNatNics) {\n        if (netuuids.indexOf(job.params.fabricNatNics[n].network_uuid) === -1) {\n            nics.push(job.params.fabricNatNics[n]);\n            netuuids.push(job.params.fabricNatNics[n].network_uuid);\n        }\n    }\n\n    async.mapSeries(nics, function (nic, next) {\n        var newTicket = {\n            scope: 'fabric_nat',\n            id: nic.network_uuid,\n            expires_at: (new Date(\n                Date.now() + 600 * 1000).toISOString())\n        };\n\n        cnapi.waitlistTicketCreate('default', newTicket, onCreate);\n\n        function onCreate(err, ticket) {\n            if (err) {\n                next(err);\n                return;\n            }\n\n            // look up ticket, ensure it's not expired or invalid\n            cnapi.waitlistTicketGet(ticket.uuid,\n                function (geterr, getticket) {\n                    if (geterr) {\n                        next(geterr);\n                        return;\n                    }\n\n                    job.params.fabricNatTickets.push({\n                        nic: nic,\n                        ticket: getticket\n                    });\n                    job.log.info(\n                        { nic: nic, ticket: getticket },\n                        'ticket status after create');\n                    next();\n                });\n        }\n    }, function (sErr) {\n        if (sErr) {\n            cb(sErr);\n        } else {\n            cb(null, 'Fabric NAT tickets acquired');\n        }\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "napi.provision_fabric_nats",
      "timeout": 120,
      "retry": 1,
      "body": "function provisionFabricNats(job, cb) {\n    if (!job.params.fabricNatTickets ||\n            job.params.fabricNatTickets.length === 0) {\n        return cb(null, 'No fabric NATs to provision');\n    }\n\n    if (!job.params.sdc_nat_pool) {\n        return cb(new Error('No fabric NAT pool configured for provisioning'));\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var napi = new sdcClients.NAPI({\n        url: napiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var natSvc;\n    var sapi = new sdcClients.SAPI({\n        log: job.log.child({ component: 'sapi' }),\n        url: sapiUrl,\n        version: '~2',\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    function releaseTicket(tErr, ticket, tCb) {\n        cnapi.waitlistTicketRelease(ticket.uuid, function (relErr) {\n            if (relErr) {\n                job.log.error({ ticket: ticket, err: relErr },\n                    'Error releasing ticket');\n            }\n\n            if (tErr) {\n                tCb(tErr);\n                return;\n            }\n\n            tCb(relErr);\n            return;\n        });\n    }\n\n    /*\n     * Provision a new NAT zone through SAPI on two networks:\n     * - the configured NAT network pool\n     * - the fabric network that needs a NAT zone\n     */\n    function provisionNatZone(tick, done) {\n        var fabricNic = tick.nic;\n\n        // If we were waiting on a ticket because another NAT zone was being\n        // provisioned and it succeeded, we don't need to provision another.\n        napi.getNetwork(fabricNic.network_uuid, function (netErr, fNet) {\n            if (netErr) {\n                return done(netErr);\n            }\n\n            if (fNet.gateway_provisioned) {\n                job.log.debug({ ticket: tick.ticket.uuid, net: fNet },\n                    'Network already has gateway provisioned');\n                tick.gateway_provisioned = true;\n                return releaseTicket(null, tick.ticket, done);\n            }\n\n            var instParams = {\n                metadata: {\n                    'com.joyent:ipnat_subnet': fNet.subnet\n                },\n                params: {\n                    alias: 'nat-' + fabricNic.network_uuid,\n                    internal_metadata: {\n                        'com.joyent:ipnat_owner': job.params.owner_uuid\n                    },\n                    networks: [\n                        {\n                            uuid: job.params.sdc_nat_pool,\n                            primary: true,\n                            allow_ip_spoofing: true\n                        },\n                        {\n                            uuid: fabricNic.network_uuid,\n                            ip: fabricNic.gateway,\n                            allow_ip_spoofing: true\n                        }\n                    ],\n                    ticket: tick.ticket.uuid\n                }\n            };\n\n            sapi.createInstanceAsync(natSvc, instParams,\n                    function _afterSapiProv(createErr, inst) {\n                if (createErr) {\n                    return releaseTicket(createErr, tick.ticket, done);\n                }\n\n                job.log.info({ instance: inst, natSvc: natSvc },\n                    'Created NAT instance');\n\n                tick.job_uuid = inst.job_uuid;\n                tick.vm_uuid = inst.uuid;\n                return done();\n            });\n        });\n    }\n\n    sapi.listServices({ name: 'nat' }, function (sapiErr, svcs) {\n        if (sapiErr) {\n            return cb(sapiErr);\n        }\n\n        if (!svcs || svcs.length === 0) {\n            return cb(new Error('No \"nat\" service found in SAPI'));\n        }\n\n        if (svcs.length > 1) {\n            return cb(new Error('More than one \"nat\" service found in SAPI'));\n        }\n\n        natSvc = svcs[0].uuid;\n        job.log.info({ svc: natSvc, svcs: svcs }, 'svcs');\n\n        async.forEach(job.params.fabricNatTickets, function (tick, next) {\n            if (tick.ticket.status === 'active') {\n                return provisionNatZone(tick, next);\n            }\n\n            cnapi.waitlistTicketWait(tick.ticket.uuid,\n                    function _afterWait(tErr) {\n                if (tErr) {\n                    next(tErr);\n                } else {\n                    provisionNatZone(tick, next);\n                }\n            });\n\n        }, function (aErr) {\n            if (aErr) {\n                cb(aErr);\n            } else {\n                cb(null, 'Provisioned fabric NATs');\n            }\n        });\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "common.update_network_params",
      "timeout": 10,
      "retry": 1,
      "body": "function updateNetworkParams(job, cb) {\n    var toAdd = job.params.add_nics;\n    if (toAdd === undefined) {\n        cb('add_nics are required');\n        return;\n    }\n\n    // From the list of oldResolvers append the new ones\n    var i, j, nic, resolver;\n    var resolvers = job.params.oldResolvers || [];\n    job.log.info(job.params.oldResolvers, 'oldResolvers');\n\n    for (i = 0; i <  toAdd.length; i++) {\n        nic = toAdd[i];\n\n        if (nic['resolvers'] !== undefined &&\n            Array.isArray(nic['resolvers'])) {\n            for (j = 0; j < nic['resolvers'].length; j++) {\n                resolver = nic['resolvers'][j];\n                if (resolvers.indexOf(resolver) === -1) {\n                    resolvers.push(resolver);\n                }\n            }\n        }\n    }\n\n    if (job.params.wantResolvers && resolvers.length !== 0) {\n        job.params.resolvers = resolvers;\n    }\n\n    var napi = new sdcClients.NAPI({\n        url: napiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var params = {\n        belongs_to_uuid: job.params.uuid || job.params.vm_uuid,\n        belongs_to_type: 'zone'\n    };\n\n    napi.listNics(params, function (err, res) {\n        if (err) {\n            cb(err);\n            return;\n        }\n\n        var routes = {};\n        var allNics = res.concat(toAdd);\n        for (i = 0; i < allNics.length; i++) {\n            nic = allNics[i];\n\n            if (nic['routes'] !== undefined &&\n                typeof (nic['routes']) === 'object') {\n                for (var r in nic['routes']) {\n                    if (!routes.hasOwnProperty(r)) {\n                        routes[r] = nic['routes'][r];\n                    }\n                }\n            }\n        }\n\n        if (Object.keys(routes).length !== 0) {\n            job.params.set_routes = routes;\n        }\n\n        return cb(null, 'Added network parameters to payload');\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "cnapi.acquire_vm_ticket",
      "timeout": 10,
      "retry": 1,
      "body": "function acquireVMTicket(job, cb) {\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var server_uuid = job.params.server_uuid;\n    var newTicket = {\n        scope: 'vm',\n        id: job.params.vm_uuid,\n        expires_at: (new Date(\n            Date.now() + 600 * 1000).toISOString()),\n        action: job.action\n    };\n\n    if (job.action === 'provision') {\n        newTicket.extra = {\n            workflow_job_uuid: job.uuid,\n            owner_uuid: job.params.owner_uuid,\n            max_physical_memory: job.params.max_physical_memory,\n            cpu_cap: job.params.cpu_cap,\n            quota: job.params.quota,\n            brand: job.params.brand,\n            disks: job.params.disks\n        };\n\n        if (['bhyve', 'kvm'].indexOf(job.params.brand) !== -1 &&\n            job.params.image) {\n\n            newTicket.extra.image_size = job.params.image.image_size;\n        }\n    }\n\n    cnapi.waitlistTicketCreate(server_uuid, newTicket, onCreate);\n\n    function onCreate(err, ticket) {\n        if (err) {\n            cb(err);\n            return;\n        }\n\n        // look up ticket, ensure it's not expired etc\n        cnapi.waitlistTicketGet(ticket.uuid,\n            function (geterr, getticket) {\n                if (geterr) {\n                    cb(geterr);\n                    return;\n                }\n                job.ticket = getticket;\n                job.log.info(\n                    { ticket: getticket }, 'ticket status after wait');\n                cb();\n            });\n    }\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "cnapi.wait_on_vm_ticket",
      "timeout": 120,
      "retry": 1,
      "body": "function waitOnVMTicket(job, cb) {\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    var ticket = job.ticket;\n\n    if (ticket.status === 'active') {\n        cb();\n        return;\n    }\n    cnapi.waitlistTicketWait(job.ticket.uuid, cb);\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "cnapi.wait_for_fabric_nat_provisions",
      "timeout": 600,
      "retry": 1,
      "body": "function waitForFabricNatProvisions(job, cb) {\n    if (!job.params.fabricNatTickets ||\n            job.params.fabricNatTickets.length === 0) {\n        return cb(null, 'No fabric NATs provisioned');\n    }\n\n    // Filter out tickets that didn't end up needing a gateway provisioned\n    var toWaitFor = job.params.fabricNatTickets.filter(function (t) {\n        return !t.gateway_provisioned;\n    });\n\n    if (toWaitFor.length === 0) {\n        return cb(null, 'No fabric NAT provisions left to wait for');\n    }\n\n    var vmapi = new sdcClients.VMAPI({\n        url: vmapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    function checkVm(tick, done) {\n        var uuid = tick.vm_uuid;\n        vmapi.getVm({ uuid: uuid }, onVmapi);\n\n        function onVmapi(err, vm, req, res) {\n            if (err) {\n                cb(err);\n\n            } else if (vm.state === 'running') {\n                done();\n\n            } else if (vm.state === 'failed') {\n                done(new Error(\n                        'NAT zone \"' + vm.uuid + '\" failed to provision'));\n\n            } else {\n                setTimeout(checkVm, 1000, tick, done);\n            }\n        }\n    }\n\n    async.forEach(toWaitFor, checkVm, function (aErr) {\n        if (aErr) {\n            cb(aErr);\n        } else {\n            cb(null, 'Fabric NATs running');\n        }\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "cnapi.update_vm",
      "timeout": 10,
      "retry": 1,
      "body": "function zoneAction(job, cb) {\n    if (job.params['skip_zone_action']) {\n        cb(null, 'Skipping zoneAction');\n        return;\n    }\n\n    if (!job.endpoint) {\n        cb('No CNAPI endpoint provided');\n        return;\n    }\n\n    if (!job.requestMethod) {\n        cb('No HTTP request method provided');\n        return;\n    }\n\n    // Not using sdc-clients to allow calling generic POST actions without\n    // explicitly saying: startVm, stopVm, etc\n    var cnapi = restify.createJsonClient({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    // Use payload when available\n    var payload = job.params.payload || job.params;\n\n    function callback(err, req, res, task) {\n        if (err) {\n            cb(err);\n        } else {\n            job.taskId = task.id;\n            cb(null, 'Task id: ' + task.id + ' queued to CNAPI!');\n        }\n    }\n\n    if (job.requestMethod == 'post') {\n        cnapi.post(job.endpoint, payload, callback);\n    } else if (job.requestMethod == 'put') {\n        cnapi.put(job.endpoint, payload, callback);\n    } else if (job.requestMethod == 'del') {\n        cnapi.del(job.endpoint, callback);\n    } else {\n        cb('Unsupported requestMethod: \"' + job.requestMethod + '\"');\n    }\n}",
      "modules": {
        "restify": "restify"
      }
    },
    {
      "name": "cnapi.wait_task",
      "timeout": 120,
      "retry": 1,
      "body": "function waitTask(job, cb) {\n    if (job.params['skip_zone_action']) {\n        cb(null, 'Skipping waitTask');\n        return;\n    }\n\n    if (!job.taskId) {\n        cb('No taskId provided');\n        return;\n    }\n\n    if (!cnapiUrl) {\n        cb('No CNAPI URL provided');\n        return;\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    cnapi.waitTask(job.taskId, {}, onTask);\n\n    function onTask(err, task) {\n        if (err) {\n            if (err.statusCode === 404) {\n                // fallback to pollTask\n                cnapi.pollTask(job.taskId, {}, function (pollerr, polltask) {\n                    // Make sure loops cannot happen\n                    if (pollerr && pollerr.statusCode === 404) {\n                        cb(pollerr);\n                        return;\n                    }\n                    onTask(pollerr, polltask);\n                });\n                return;\n            }\n            cb(err);\n        } else if (task && task.status == 'failure') {\n            cb(getErrorMesage(task));\n        } else if (task && task.status == 'complete') {\n            // Tasks that modify VM state should add a .vm to the task\n            // with something like \"self.finish({ vm: machine });\"\n            if (task.history && task.history.length > 0 &&\n                task.history[0].name === 'finish' &&\n                task.history[0].event &&\n                task.history[0].event.vm) {\n\n                job.finished_vm = task.history[0].event.vm;\n                job.log.debug({vm_uuid: job.finished_vm.uuid},\n                    'finish() returned VM');\n            }\n\n            cb(null, 'Job succeeded!');\n        } else {\n            cb(new Error('unexpected task status, ' + task.status));\n        }\n    }\n\n    function getErrorMesage(task) {\n        var message;\n        var details = [];\n\n        if (task.history !== undefined && task.history.length) {\n            for (var i = 0; i < task.history.length; i++) {\n                var event = task.history[i];\n                if (event.name && event.name === 'error' && event.event &&\n                    event.event.error) {\n                    var err = event.event.error;\n                    if (typeof (err) === 'string') {\n                        message = err;\n                        if (event.event.details && event.event.details.error) {\n                            message += ', ' + event.event.details.error;\n                        }\n                    } else {\n                        message = err.message;\n                    }\n                } else if (event.name && event.name === 'finish' &&\n                    event.event && event.event.log && event.event.log.length) {\n                    for (var j = 0; j < event.event.log.length; j++) {\n                        var logEvent = event.event.log[j];\n                        if (logEvent.level && logEvent.level === 'error') {\n                            details.push(logEvent.message);\n                        }\n                    }\n                }\n            }\n        }\n\n        // Apparently the task doesn't have any message for us...\n        if (message === undefined) {\n            message = 'Unexpected error occured';\n        } else if (details.length) {\n            message += ': ' + details.join(', ');\n        }\n\n        return message;\n    }\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "vmapi.check_updated",
      "timeout": 90,
      "retry": 1,
      "body": "function checkUpdated(job, cb) {\n    if (!job.params['vm_uuid']) {\n        cb('No VM UUID provided');\n        return;\n    }\n\n    if (!vmapiUrl) {\n        cb('No VMAPI URL provided');\n        return;\n    }\n\n    if (!job.params['last_modified']) {\n        cb('No VM last_modified timestamp provided');\n        return;\n    }\n\n    var oldDate = new Date(job.params['last_modified']);\n    var vmapi = new sdcClients.VMAPI({\n        url: vmapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    // Repeat checkVm until VM data is updated\n    checkVm();\n\n    function checkVm() {\n        vmapi.listVms({ uuid: job.params['vm_uuid'] }, onVmapi);\n\n        function onVmapi(err, vms, req, res) {\n            if (err) {\n                cb(err);\n            } else if (vms.length && (vms[0].uuid == job.params['vm_uuid'])) {\n                var newDate = new Date(vms[0]['last_modified']);\n\n                if (newDate > oldDate) {\n                    cb(null, 'VM data has been updated');\n                } else {\n                    if (job.timeToDie) {\n                        job.log.error('checkUpdated.checkVm.onVmapi called '\n                            + 'after task completion, breaking loop');\n                        return;\n                    }\n                    setTimeout(checkVm, 1000);\n                }\n            }\n        }\n    }\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "vmapi.put_vm",
      "timeout": 60,
      "retry": 1,
      "body": "function putVm(job, cb) {\n    var vmapi;\n\n    if (job.params.do_not_inventory) {\n        cb(null,\n            'VM has do_not_inventory set, no need to putVm to VMAPI');\n        return;\n    }\n\n    /*\n     * Checks (polls) the state of a machine in VMAPI. It is used for provisions\n     * and VM actions such as reboot and shutdown.\n     *\n     * IMPORTANT: this function an all uses of job.expects are deprecated and\n     *            will be removed in a future version after everyone is updated\n     *            past the old agent tasks that don't pass back the VMs. It is\n     *            being replaced with the putVm function and is now only called\n     *            from there.\n     */\n    function checkState(_job, _cb) {\n        if (_job.params['skip_zone_action']) {\n            _cb(null, 'Skipping checkState');\n            return;\n        }\n\n        // For now don't fail the job if this parameter is not present\n        if (!_job.expects) {\n            _cb(null, 'No \\'expects\\' state parameter provided');\n            return;\n        }\n\n        if (!_job.params['vm_uuid']) {\n            _cb('No VM UUID provided');\n            return;\n        }\n\n        if (!vmapiUrl) {\n            _cb('No VMAPI URL provided');\n            return;\n        }\n\n        var _vmapi = new sdcClients.VMAPI({\n            url: vmapiUrl,\n            headers: { 'x-request-id': _job.params['x-request-id'] }\n        });\n\n        // Repeat checkVm until VM data is updated\n        checkVm();\n\n        function checkVm() {\n            _vmapi.getVm({ uuid: _job.params['vm_uuid'] }, onVmapi);\n\n            function onVmapi(err, vm, req, res) {\n                if (err) {\n                    _cb(err);\n                } else if (vm.state == _job.expects) {\n                    _cb(null, 'VM is now ' + _job.expects);\n                } else {\n                    if (_job.timeToDie) {\n                        _job.log.error('checkState.checkVm.onVmapi called after'\n                            + ' task completion, breaking loop');\n                        return;\n                    }\n                    setTimeout(checkVm, 1000);\n                }\n            }\n        }\n    }\n\n    if (!job.finished_vm) {\n        job.log.warn({req_id: job.params['x-request-id']},\n            'putVM() called but job.finished_vm is missing');\n\n        checkState(job, cb);\n        //\n        // When checkState is removed:\n        //\n        // cb(null, 'job has no finished_vm, nothing to post to VMAPI');\n        return;\n    }\n\n    if (!vmapiUrl) {\n        cb(new Error('No VMAPI URL provided'));\n        return;\n    }\n\n    job.log.debug({vmobj: job.finished_vm}, 'putVM() putting VM to VMAPI');\n\n    //\n    // Borrowed from vm-agent lib/vmapi-client.js\n    //\n    // DO NOT TRY THIS AT HOME!\n    //\n    // afaict the reason sdcClients does not have a putVm function in the first\n    // place is that this is not something API clients should generally be\n    // doing. WE need to do it, and vm-agent needs to do it, but other clients\n    // should not be doing it unless they're absolutely sure that what they're\n    // PUTing is the current state.\n    //\n    // We know that here because cn-agent tasks just did a VM.load for us.\n    //\n    sdcClients.VMAPI.prototype.putVm = function (vm, callback) {\n        var log = job.log;\n        var opts = { path: '/vms/' + vm.uuid };\n\n        this.client.put(opts, vm, function (err, req, res) {\n            if (err) {\n                log.error(err, 'Could not update VM %s', vm.uuid);\n                return callback(err);\n            }\n\n            log.info('VM (uuid=%s, state=%s, last_modified=%s) updated @ VMAPI',\n                vm.uuid, vm.state, vm.last_modified);\n            return callback();\n        });\n    };\n\n    vmapi = new sdcClients.VMAPI({\n        log: job.log,\n        url: vmapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    vmapi.putVm(job.finished_vm, function (err) {\n        if (err) {\n            cb(err);\n            return;\n        }\n\n        cb(null, 'put VM ' + job.finished_vm.uuid + ' to VMAPI');\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "fwapi.update",
      "timeout": 10,
      "retry": 1,
      "body": "function updateFwapi(job, cb) {\n    var fwapi = new sdcClients.FWAPI({\n        url: fwapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    var jobParams = job.params.payload || job.params;\n    var type;\n    var update = {};\n    var vmProps = ['add_nics', 'firewall_enabled', 'nics', 'remove_ips',\n        'remove_nics', 'remove_tags', 'set_tags', 'tags'];\n\n    if (job.params.task === 'provision') {\n        type = 'vm.add';\n    } else {\n        type = (job.params.task === 'destroy') ? 'vm.delete' : 'vm.update';\n    }\n\n    vmProps.forEach(function (prop) {\n        if (jobParams.hasOwnProperty(prop)) {\n            update[prop] = jobParams[prop];\n        }\n    });\n\n    job.log.info({ jobParams: jobParams, update: update }, 'update params');\n\n    if (Object.keys(update).length === 0 && job.params.task !== 'destroy') {\n        cb(null, 'No properties affecting FWAPI found: not updating');\n        return;\n    }\n\n    update.owner_uuid = jobParams.owner_uuid || job.params.owner_uuid;\n    update.server_uuid = jobParams.server_uuid || job.params.server_uuid;\n    update.type = type;\n    update.uuid = jobParams.uuid || jobParams.vm_uuid || job.params.vm_uuid;\n\n    fwapi.createUpdate(update, function (err, obj) {\n        if (err) {\n            job.log.warn(err, 'Error sending update to FWAPI');\n            cb(null, 'Error updating FWAPI');\n            return;\n        }\n\n        cb(null, 'Updated FWAPI with update UUID: ' + obj.update_uuid);\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "cnapi.release_vm_ticket",
      "timeout": 60,
      "retry": 1,
      "body": "function releaseVMTicket(job, cb) {\n    if (!job.ticket) {\n        return cb();\n    }\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    cnapi.waitlistTicketRelease(job.ticket.uuid, function (err) {\n        if (err) {\n            job.log.warn({err: err, ticket: job.ticket},\n                'error releasing CNAPI waitlist VM ticket');\n        }\n        cb(err);\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "cnapi.release_fabric_nat_ticket",
      "timeout": 60,
      "retry": 1,
      "body": "function releaseFabricTicket(job, cb) {\n    if (!job.params.ticket) {\n        return cb(null, 'No ticket to release');\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    cnapi.waitlistTicketRelease(job.params.ticket, function (err) {\n        if (err) {\n            if (err.code === 'ResourceNotFound') {\n                cb(null, 'Ticket released');\n            } else {\n                cb(err);\n            }\n\n            return;\n        }\n\n        cb(null, 'Released ticket ' + job.params.ticket);\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    }
  ],
  "timeout": 300,
  "onerror": [
    {
      "name": "napi.cleanup_nics",
      "timeout": 10,
      "retry": 1,
      "body": "function cleanupNics(job, cb) {\n    // If this is false it means that cnapi.pollTask succeeded, so the VM exists\n    // physically wether its provision failed or not\n    if (job.markAsFailedOnError === false) {\n        return cb(null, 'markAsFailedOnError was set to false, ' +\n            'won\\'t cleanup VM NICs');\n    }\n\n    var macs = job.params.macs;\n\n    if (!macs) {\n            /*\n             * filteredNetworks.nics will contain any pre provisioned NICs. If\n             * the workflow fails early enough the other job.params fields will\n             * not yet have been populated yet, but we still have some NICs that\n             * need to be removed.\n             */\n            var nics = job.params['add_nics'] || job.params['nics'] ||\n                job.params.filteredNetworks.nics;\n\n            if (!nics) {\n                return cb(null, 'No MACs given, and no NICs were provisioned');\n            }\n\n            macs = nics.map(function (nic) { return nic.mac; });\n    }\n\n    var napi = new sdcClients.NAPI({\n        url: napiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    async.mapSeries(macs, function (mac, next) {\n        napi.deleteNic(mac, next);\n    }, function (err) {\n        if (err) {\n            cb(err);\n        } else {\n            cb(null, 'NICs removed');\n        }\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients",
        "async": "async"
      }
    },
    {
      "name": "on_error.release_vm_ticket",
      "modules": {
        "sdcClients": "sdc-clients"
      },
      "body": "function releaseVMTicket(job, cb) {\n    if (!job.ticket) {\n        return cb();\n    }\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    cnapi.waitlistTicketRelease(job.ticket.uuid, function (err) {\n        if (err) {\n            job.log.warn({err: err, ticket: job.ticket},\n                'error releasing CNAPI waitlist VM ticket');\n        }\n        cb(err);\n    });\n}"
    },
    {
      "name": "cnapi.release_fabric_nat_ticket",
      "timeout": 60,
      "retry": 1,
      "body": "function releaseFabricTicket(job, cb) {\n    if (!job.params.ticket) {\n        return cb(null, 'No ticket to release');\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    cnapi.waitlistTicketRelease(job.params.ticket, function (err) {\n        if (err) {\n            if (err.code === 'ResourceNotFound') {\n                cb(null, 'Ticket released');\n            } else {\n                cb(err);\n            }\n\n            return;\n        }\n\n        cb(null, 'Released ticket ' + job.params.ticket);\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    },
    {
      "name": "On error",
      "body": "function (job, cb) {\n            return cb('Error executing job');\n        }"
    }
  ],
  "oncancel": [
    {
      "name": "on_cancel.release_vm_ticket",
      "modules": {
        "sdcClients": "sdc-clients"
      },
      "body": "function releaseVMTicket(job, cb) {\n    if (!job.ticket) {\n        return cb();\n    }\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n    cnapi.waitlistTicketRelease(job.ticket.uuid, function (err) {\n        if (err) {\n            job.log.warn({err: err, ticket: job.ticket},\n                'error releasing CNAPI waitlist VM ticket');\n        }\n        cb(err);\n    });\n}"
    },
    {
      "name": "cnapi.release_fabric_nat_ticket",
      "timeout": 60,
      "retry": 1,
      "body": "function releaseFabricTicket(job, cb) {\n    if (!job.params.ticket) {\n        return cb(null, 'No ticket to release');\n    }\n\n    var cnapi = new sdcClients.CNAPI({\n        url: cnapiUrl,\n        headers: { 'x-request-id': job.params['x-request-id'] }\n    });\n\n    cnapi.waitlistTicketRelease(job.params.ticket, function (err) {\n        if (err) {\n            if (err.code === 'ResourceNotFound') {\n                cb(null, 'Ticket released');\n            } else {\n                cb(err);\n            }\n\n            return;\n        }\n\n        cb(null, 'Released ticket ' + job.params.ticket);\n    });\n}",
      "modules": {
        "sdcClients": "sdc-clients"
      }
    }
  ],
  "workflow_uuid": "cfbae18e-059f-4f05-a566-50886365a5a5",
  "created_at": "2019-07-03T13:19:16.399Z",
  "onerror_results": [
    {
      "result": "NICs removed",
      "error": "",
      "name": "napi.cleanup_nics",
      "started_at": "2019-07-03T13:21:21.405Z",
      "finished_at": "2019-07-03T13:21:21.896Z"
    },
    {
      "result": "OK",
      "error": "",
      "name": "on_error.release_vm_ticket",
      "started_at": "2019-07-03T13:21:22.047Z",
      "finished_at": "2019-07-03T13:21:22.234Z"
    },
    {
      "result": "No ticket to release",
      "error": "",
      "name": "cnapi.release_fabric_nat_ticket",
      "started_at": "2019-07-03T13:21:22.384Z",
      "finished_at": "2019-07-03T13:21:22.416Z"
    },
    {
      "result": "",
      "error": "Error executing job",
      "name": "On error",
      "started_at": "2019-07-03T13:21:22.537Z",
      "finished_at": "2019-07-03T13:21:22.568Z"
    }
  ],
  "oncancel_results": [],
  "started": 1562159957570,
  "endpoint": "/servers/38363837-3034-324d-3238-353130313433/vms/f0e54000-725a-4224-a0a3-c4d7138c4f7d/update",
  "requestMethod": "post",
  "action": "add_nics",
  "serverNicTags": "[\"admin\",\"external\",\"sdc_underlay\",\"sdc_overlay\"]",
  "ticket": "{\"action\":\"add_nics\",\"created_at\":\"2019-07-03T13:19:20.803Z\",\"expires_at\":\"2019-07-03T13:29:20.731Z\",\"extra\":{},\"id\":\"f0e54000-725a-4224-a0a3-c4d7138c4f7d\",\"scope\":\"vm\",\"server_uuid\":\"38363837-3034-324d-3238-353130313433\",\"status\":\"active\",\"updated_at\":\"2019-07-03T13:19:20.916Z\",\"uuid\":\"20f747e8-22ba-cd06-f216-e3ee50cf7bba\"}",
  "taskId": "d94a66ee-2640-caae-b8c2-ab554f27807e",
  "finished_vm": "{\"zonename\":\"f0e54000-725a-4224-a0a3-c4d7138c4f7d\",\"autoboot\":true,\"brand\":\"lx\",\"limit_priv\":\"default\",\"v\":1,\"create_timestamp\":\"2019-07-03T13:17:03.470Z\",\"image_uuid\":\"3dbbdcca-2eab-11e8-b925-23bf77789921\",\"cpu_shares\":128,\"max_lwps\":4000,\"max_msg_ids\":4096,\"max_sem_ids\":4096,\"max_shm_ids\":4096,\"max_shm_memory\":2048,\"zfs_io_priority\":128,\"max_physical_memory\":2048,\"max_locked_memory\":2048,\"max_swap\":8192,\"cpu_cap\":200,\"billing_id\":\"5010c1e3-e2d6-e57c-bc8e-fb1fb19dee82\",\"tmpfs\":2048,\"hostname\":\"addnictest\",\"dns_domain\":\"inst.bruce-dev.us-east-1.bdf-cloud.iqvia.net\",\"archive_on_delete\":true,\"kernel_version\":\"3.10.0\",\"alias\":\"addnictest\",\"nics\":[{\"interface\":\"eth0\",\"mac\":\"90:b8:d0:96:25:47\",\"vlan_id\":800,\"nic_tag\":\"external\",\"gateway\":\"10.45.136.1\",\"gateways\":[\"10.45.136.1\"],\"netmask\":\"255.255.255.0\",\"ip\":\"10.45.136.195\",\"ips\":[\"10.45.136.195/24\"],\"network_uuid\":\"a1f3f50c-884c-4ff5-92fa-fcb8ad30162c\",\"mtu\":1500,\"primary\":true},{\"interface\":\"eth1\",\"mac\":\"90:b8:d0:b4:20:bd\",\"vlan_id\":100,\"nic_tag\":\"sdc_overlay/13954390\",\"gateway\":\"10.1.1.1\",\"gateways\":[\"10.1.1.1\"],\"netmask\":\"255.255.255.0\",\"ip\":\"10.1.1.95\",\"ips\":[\"10.1.1.95/24\"],\"network_uuid\":\"0a63f067-d721-435e-941a-724776d69c91\",\"mtu\":8500}],\"owner_uuid\":\"2331453d-5310-42a1-dd63-92dcc4004f0d\",\"resolvers\":[\"10.45.137.14\",\"10.45.137.15\"],\"uuid\":\"f0e54000-725a-4224-a0a3-c4d7138c4f7d\",\"zone_state\":\"running\",\"zonepath\":\"/zones/f0e54000-725a-4224-a0a3-c4d7138c4f7d\",\"hvm\":false,\"zoneid\":155,\"zonedid\":3487,\"last_modified\":\"2019-07-03T13:19:23.000Z\",\"firewall_enabled\":false,\"server_uuid\":\"38363837-3034-324d-3238-353130313433\",\"datacenter_name\":\"us-east-1\",\"platform_buildstamp\":\"20190619T233708Z\",\"state\":\"running\",\"boot_timestamp\":\"2019-07-03T13:17:15.000Z\",\"init_restarts\":0,\"pid\":601859,\"customer_metadata\":{\"root_authorized_keys\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/ bruce.smith@Bruces-MacBook-Pro.local\\n\"},\"internal_metadata\":{},\"routes\":{},\"tags\":{},\"quota\":50,\"zfs_root_compression\":\"on\",\"zfs_root_recsize\":131072,\"zfs_filesystem\":\"zones/f0e54000-725a-4224-a0a3-c4d7138c4f7d\",\"zpool\":\"zones\",\"snapshots\":[]}",
  "elapsed": 125.124,
  "uuid": "34c837b1-5502-46d2-9ced-20a10c88b325"
}

@twhiteman
Copy link
Contributor

Oh, my add_nics workflow job failed:

    {
      "result": "Task id: 85895777-819b-6140-9c23-df44532bd1e6 queued to CNAPI!",
      "error": "",
      "name": "cnapi.update_vm",
      "started_at": "2019-07-03T23:51:30.136Z",
      "finished_at": "2019-07-03T23:51:30.581Z"
    },
    {
      "result": "",
      "error": "vmadm.reboot error: vmadm exited with code: 1 signal: null",
      "name": "cnapi.wait_task",
      "started_at": "2019-07-03T23:51:30.714Z",
      "finished_at": "2019-07-03T23:51:35.849Z"
    }

so it failed to reboot correctly and prematurely ended the add_nics workflow (so the workflow then runs the on_error handler chain).

The add_nics on_error workflow is meant to run the napi.cleanup_nics cleanup code (to remove those params.add_nics nics), but in my case (and your case too) I see it says it ran the cleanup (napi delete of everything in params.add_nics) and I also see that napi received this delete and responsed with a successful 204 (No Content)... yet my vm still has that just added nic still on it (according to vmadm and vmapi), so I really don't know why those nics are still there and/or why it's working this way.

More digging is needed.

@Smithx10
Copy link
Author

Smithx10 commented Jul 4, 2019

@twhiteman What is the reasoning for rebooting an instance for a nic add and delete? I thought SmartOS could add NICs without bounce.

@twhiteman
Copy link
Contributor

Adding or removing a NIC has always needed a bounce, as far as I remember. I'm not sure of the reasoning, but it's mentioned in the CloudAPI docs:
https://apidocs.joyent.com/cloudapi/#AddNic

@twhiteman
Copy link
Contributor

twhiteman commented Jul 5, 2019 •
edited
Loading

After updating my platform (for the reboot bug, aka OS-5176), I'm now getting successful workflow jobs, but still no sign of the issue you've seen:

Waiting for vm to start
Waiting for vm to start
Waiting for vm to start
Waiting for vm to start
Waiting for vm to start
Waiting for vm to start
Vm is running
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: provisioning
Nic state: running

and the jobs:

# sdc-vmapi /vms/6118fb68-4b14-4dad-eaa9-d3d225f31f98/jobs | json -Hga created_at name execution | sort
2019-07-05T22:08:36.516Z provision-8.2.1 succeeded
2019-07-05T22:08:51.363Z add-nics-7.3.1 succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@twhiteman @Smithx10