/
PerfSONAR pScheduler (Tests Scheduling)

PerfSONAR pScheduler (Tests Scheduling)

Owned by UKY CCS RCD
Jun 21, 2024


Basic Tests

  • Throughput
    • Tests amount of data that can be transferred over a period of time
    • Managed by pScheduler
      • iperf3 (default)
      • iperf2
      • nttcp
  • Ping (Round-Trip Latency)
    • Tests round-trip time and other statistics of packets
    • Managed by pScheduler
    • Similar data to one-way latency
      • Less granularity
      • No regard to whether values occurred on the forward or reverse path
      • Much more likely to be supported by other sites
  • One-way Latency
    • Like ping, but measures forward and backward paths separately
    • Uses OWAMP client
    • Runs constantly sending several packets each second
    • May see anomalies such as a sudden burst of loss when throughput tests run beside it
      • Best to run these latency tests on a different interface/host to throughput tests
  • Traceroute
    • Periodically measures path between two destinations
    • Will be added every time any other type of test is added by default
    • Useful for identifying changes in route that may affect other tests
    • managed by pScheduler
      • tracepath (default)
      • defaults to traceroute if above not installed
      • can also choose paris-traceroute

Configuring Regular Tests via the Web UI

  1. Click the "Configuration" button in the top right-hand corner to configure tests with other hosts.

     










         2. Click on "Tests"


3. Click on "+ Test" to add a test.

4. Select the type of test.



5. Enter the test name/description and add Test Members

6. Enter Hostname/IP and Host description (optional) and click add host.



7. Click OK.

At this point, your PerfSONAR node will start testing with the hosts added and you should be able to see the results in the "Test Results" section of your toolkit homepage.

Configuring Regular Tests via a Config File

pSConfig pScheduler Agent

This is a service that runs on a perfSONAR node. Its job is to read pSConfig templates and generate a set of pScheduler tasks.
It reads the templates and runs

  • On agent startup
  • Within an amount of time of a change to a local configuration file
    • Defaults to 1 minute, but can be modified
  • On a fixed interval after the last run
    • Defaults to every 1 hour

Steps the agent performs

  1. Read templates and optionally apply local modifications
    • Templates are[ |https://en.wikipedia.org/wiki/JSON]JSON files
    • Transformations
      • Include default archives (for logging)
      • Transforming JSON directly
  2. Determines pScheduler tasks to schedule
  3. Communicate with pScheduler to create tasks

The agent is a standard Linux service, so it can be controlled with the systemctl command.

Configuration Templates

    1. Create a configuration file.
      1. There are templates of these kinds of files as /usr/share/doc/perfsonar/psconfig/example-X-NAME.json
      2. To use the template below; edit the following fields
        1. Under the "archives" section, replace archive.perfsonar.net with the hostname/IP address of a server that can store test data in a service called Esmond. The hostname/IP address of this perfSONAR works if no other server has been explicitly made for this.
        2. Under the "addresses" section, replace the items host-X.perfsonar.net with the hostname/IP addresses of one of the perfSONAR nodes you wish to include in testing.
        3. Under the "addresses" section, similarly replace the items host-X.perfsonar.net with a name for the perfSONAR nodes. This should be the hostname of the perfSONAR node preferably, but in principle any name will work.
        4. To add more nodes to these tests, add another "host-X.perfsonar.net": { "address": "host-X.perfsonar.net" }, and fill in accordingly.
      3. Use the following command to verify validity

        1. it will print an error message if the file is not formatted correctly or the JSON file if it is.

          jq . FILE.json

          FILE.json
          {
    "archives": {
        "example-archive-central": {
            "archiver": "esmond",
            "data": {
                "url": "https://archive.perfsonar.net/esmond/perfsonar/archive/",
                "measurement-agent": "{% scheduled_by_address %}"
            }
        }
    },

    "addresses": {
        "host-a.perfsonar.net": { "address": "host-a.perfsonar.net" },
        "host-b.perfsonar.net": { "address": "host-b.perfsonar.net" },
        "host-c.perfsonar.net": { "address": "host-c.perfsonar.net" }
    },

    "groups": {
         "example-group-mesh": {
            "type": "mesh",
            "addresses": [
                { "name": "host-a.perfsonar.net" },
                { "name": "host-b.perfsonar.net" },
                { "name": "host-c.perfsonar.net" }
            ]
        }
    },

    "tests": {
        "example-test-throughput": {
            "type": "throughput",
            "spec": {
                "source": "{% address[0] %}",
                "dest": "{% address[1] %}",
                "duration": "PT30S"
            }
        },
        "example-test-latencybg": {
            "type": "latencybg",
            "spec": {
                "source": "{% address[0] %}",
                "dest": "{% address[1] %}"
            }
        },
        "example-test-trace": {
            "type": "trace",
            "spec": {
                "source": "{% address[0] %}",
                "dest": "{% address[1] %}"
            }
        }
    },

    "schedules": {
        "example-schedule-PT4H": {
            "repeat": "PT4H",
            "sliprand": true,
            "slip": "PT4H"
        },
        "example-schedule-PT10M": {
            "repeat": "PT10M",
            "sliprand": true,
            "slip": "PT10M"
        }
    },


    "tasks": {
        "example-task-throughput": {
            "group": "example-group-mesh",
            "test": "example-test-throughput",
            "schedule": "example-schedule-PT4H",
            "archives": [ "example-archive-central" ]
        },
        "example-task-latencybg": {
            "group": "example-group-mesh",
            "test": "example-test-latencybg",
            "archives": [ "example-archive-central" ]
        },
        "example-task-trace": {
            "group": "example-group-mesh",
            "test": "example-test-trace",
            "schedule": "example-schedule-PT10M",
            "archives": [ "example-archive-central" ]
        }
    }
}
    2. Distribute the configuration file to every perfSONAR node mentioned in it. There are a couple of methods to do this
      1. You can manually add the file to the /etc/perfsonar/psconfig/pscheduler.d/ directory of each node.
      2. You can use the following commands to avoid manually duplicating the file.

      1. On a perfSONAR node that has a copy of the file, run the following command and copy the URL in the output of the command.

        psconfig publish FILE

      2. Run the following command on all the perfSONAR nodes

If the node with the original copy of the file has it in the directory mentioned in the previous method, this command is not necessary to run on it. If not, you can also use the path to that file instead of the URL as well in the following command.

psconfig remote add URL

psconfig remote Command

Below are some basic commands that are useful when you want to deal with managing remote templates on a node.

  • sudo psconfig remote add <URL or filesystem path>
    • Add a remote template
  • psconfig remote list
    • List all the remote templates used by the agent on the node
  • sudo psconfig remote delete <URL or filesystem path>
    • remove a remote template
  • psconfig remote --help
    • Get help with the command and view other options

Troubleshooting pScheduler Tests

Use the psconfig pscheduler-stats command to parse logs for information about last run

Viewing Managed pScheduler Tasks

Using the psconfig pscheduler-tasks command gives a JSON list of tasks pSConfig pScheduler agent manages

  • Note that this list is not necessarily the list of tasks created at the last run and does not guarantee tasks were successfully scheduled in pScheduler

Logs

  • Agent log: /var/log/perfsonar/psconfig-pscheduler-agent.log
    • Logs agent actions and errors
    • Least verbose
  • Task log: /var/log/perfsonar/psconfig-pscheduler-agent-tasks.log
    • Logs tasks the agent manages
    • Use to look at how pSConfig is defining tasks it gives to pScheduler
    • Used by psconfig pscheduler-tasks
  • Transaction log: /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log
    • Logs every interaction the agent has with pScheduler
    • Most verbose; useful for complex debugging issues
  • Logs use key=value fields
  • every line has GUID to allow you to connect entries between logs



For more information, visit the links below:




Center for Computational Sciences