FreeCONF is a software library you add to your application to allow it to be managed by DevOps tools. Whether you use FreeCONF or any other library that supports the same standard, they all appear the same way to the DevOps tools. This approach is similar to JMX or SNMP but much, much more powerful.

DevOps tools all provide a variety of ways to integrate applications to their tool. This is different. Here applications export a “map” of their management capabilities that DevOps tools can use to eliminate the integration step and let you immediately start automation. You can read more about how this is done or look through the examples to get an idea.

What a minute! You still have to integrate a library like FreeCONF into every application and DevOps tool. True, but you do it once for every application and once for every DevOps tool and now they all interoperate with each other forever no matter the application or the tool. This keeps software developers away from your Ansible playbooks, and DevOps engineers away from writing Terraform Go plugins (unless they want to).

FreeCONF is written in Go but work is in progress to make this available to many other computer languages. To that end, because this is a standard, you can use any software in replace of FreeCONF that implements the standard and still interoperate with all the same DevOps tools.

What sets FreeCONF apart from other solutions is its utilization of well-established IETF standards such as RESTCONF, YANG and gNMI. Without these standards, this approach would just look like another platform requiring community plugins to be useful. No plugins or repos of playbooks here!

You don’t have to be all in! Incorporate as much or as little of FreeCONF to suit your needs and use this alongside whatever you are using now.

1.2 - How does it work?

Very high-level description of how FreeCONF works

Applications that use FreeCONF become available on the network to compatible DevOps tools through auto discovery or direct list of addresses and ports. DevOps tools can pull the list of management capabilities from applications. DevOps tools do not understand what the application is but it knows how to send or get configuration, pull or push metrics, call functions or listen to alerts. As a DevOps engineer, you are still in control of every management facet the application makes available.

To illustrate this concept, consider the hypothetical scenario where an individual has developed a novel toaster service and seeks to imbue it with manageability.

Steps as a developer:

Describe the management capabilities of the toaster in a YANG file like this one..
Write and/or generate code to connect toaster software definitions described in the YANG file.

Steps as a operator:

Start toaster service within your IT infrastruture (doesn’t matter how : docker container, bare metal or physical device).
Any tool that can utilize REST APIs can perform access management operations. Tools that integrate RESTCONF-client support would likely make things easier.

For Example:

An alert tool can read toaster’s YANG file and discover there are two events exported by the toaster service: toasterOutOfBread and toasterRestocked. Operator can configure alert tool to notify operator on Slack when toaster is out of bread.
A configuration can get the current toaster configuration and then send an updated copy relying on the configuration being rejected if invalid for any reason.
A metrics tool can pull toast making metrics in JSON format and at any frequency they like. In addition, the metrics tool can parse the YANG and pass along the field descriptions to graphing system such as Grafana. Pushing metrics is possible but requires special server library support.
A toaster, web admin portal can do any of the above and also call RPC functions like toasting bread by just POSTing JSON data as input and parsing the JSON output response

1.3 - Why a standard?

Standards are the key.

Standards obviate the need for integration because both sides can communicate using a predetermined protocol. You don’t need to build your own adapter if you can plug right in!

Standards like TCP/IP allowed computers from different networks to communicate without having support for each other’s specific networking stack. SMTP did the same for electronic messages and HTML/HTTP did it for information.

Are RESTCONF, YANG and gNMI the next standard for application management? They might be, but whatever it is, with a library like FreeCONF, you don’t need to care. As new standards emerge the FreeCONF community will support it and your code doesn’t need to change.

Aren’t defacto standards good enough?

“Tool X is the defacto standard for doing this in the industry so I don’t see the need for an official standard”

Every standard, defacto or official has a life span. Defacto ones might be a few years, official ones can last decades. When a standard lasts long enough for an application to ship with support built in, it changes the game. An application with built in support does not rely on fragmented set of communities to fill the gap sometime later. When support is implemented by the application developers it tends to be complete because they know the application the best.

Additional reading

To get an understanding of the genesis and impact of standards you can read the book Where the Wizards Stay Up Late to appreciate the before and after of each standard.

1.4 - Why the project?

Why the project and how it is different

Current systems

Today

Terms

Let’s settle on the following definitions:

Management - Configuration, Metrics, Alerts, RPCs, Security
Example RPCs: Backup, drain queue, build report, take snapshot
Applications - Running software critical to your business
Examples: Mysql, HAProxy, Kubernetes, Custom Applications
Tools - Software used by DevOps to manage your applications
Examples: Ansible, Grafana, Kubernetes, Terraform
Integration - Custom scripts or software written by DevOps to allow tools to manage applications
Examples: Ansible playbooks, Terraform plugins, Bash, Python, Admin Utilities
Automation - High-level data and orchestration. The “Fun Stuff!"
Examples: KPI, Application Lifecycle handling, A.I., auto-scaling, auto-repair

Obstacles for DevOps

DevOps has to build the “Tool/App Integration” part only so they can get to the fun “Automation” part.
Every line of code in “Tool/App Integration” locks DevOps into each tool.
Applications upgrades or rollbacks often require changes to “Tool/App Integration” which require testing and change management control.
Each application is different and code reuse can be hit or miss.
Copying code from internet to jump start integration often finds incomplete, buggy unmaintained solutions. Anyway you look at it, you now own it.
When you discover a new config, metrics or alert you want to capture, it means redeploying code “Tool/App Integration” which requires testing and change management control.
Configuration often requires creating a template even when you want to change a setting or two. This template is constantly changing with each application version which means deploying a new template which means more testing and change management control.
Each and every management change requires application developers to communicate the change before support is needed and enough time to accomodate the change.
Often you need to support multiple versions of an application which means support both versions of an application in case of rollback. It also means knowing when to safely remove legacy versions.

Obstacles for Application Developers

Creating new configuration means relying on DevOps to support it first
Watching for health related events often means documenting precise logging conventions and then coordinating with DevOps. Once defined the structure is difficult to change and prone to siliently breaking on subtle application changes. Because these are alerts, you don’t find out they are broken until after your customers do.
Like health events, creating new application metrics often requires DevOps to write structured log parsers and possibly new ingestion scripts. DevOp will prioritize metrics that indicate stability over simply informative metrics and could push back in implementing at all.
If any of the above is slowing you down and you want to help, you need to learn a lot of tools, how to use them safely, securely and in a maintainable manor.
Any time you spend in DevOps is time you are not writing application code
For streaming metrics, you need to integrate client libraries into your application code locking you into that tool. This also means when developing code locally, you need to be able to disable metrics. This also means issues with metrics streaming won’t be discovered until in production.

Obstacles for Software Vendors

When you have multiple services, customers often struggle to get everything perfectly configured together. You end up building custom tools to help them. These tools are expensive to build and sometimes conflict with tools customers are already using.
Producing helper scripts for each and every management tool is a lot of work and will never meet every customer’s needs.
Applications metrics are often off the table because it means coupling your software to every conceivable metrics system available. Only option is logging and hope your customers implement log parsing. It also means never changing your logging strategy.

The ideal system!

Today

Here all the applications and tools agree to a predetermined, industry wide, management standard protocol. Tools download the applications published management manifests and operate on behalf of the DevOps teams for automation. Manifests list every configuration, metrics, alert and RPC. Once the manifest is exchanged, then tools know how to fullfill automation tasks from the tool on behalf of the DevOps engineer.

Aside from removing every obtacle list above, some more reliefs come forward:

Reliefs for DevOps

Entire configuration, metric, alert, RPC available across every appliction, infrastructure, tool and OS available ready for automation with zero effort.
Precise documentation on every management facet.
Freedom to change tools or use multiple tools as they see fit without having to change application or integration code.
Configuration is validated before being applied. If configuration is wrong, it will not apply.

Reliefs for Application Developers

Empowered to connect all application management operations without requiring any tasks from DevOps.
Freedom to use different sets of tool for different environments. Use Prometheus in local development, log to postgres in end to end testing and use AWS Active monitoring for production. All work on the same application binaries unaltered.
Generate all documentation. Documentation is still important so DevOps can decide which options they need to automate.
Your applications get deployed faster because integration scripts do not need to be updated or tested.
Safely do rollbacks and again because of no integration code to break

Reliefs for Software Vendors

All your software is compatible with every customers mananagement tools.

The catch

Everyone would have to agree on a standard.
Standard would have to be capable enough to match everyone’s needs.
After agreeing on a standard, all applications and tools would then have to implement the standard.

Let’s tackle each one.

Agreeing on a standard

Every tool and application uses a library to implement the standard. If there were multiple standards, then the libraries can abstract out the difference by supporting them all. FreeCONF supports both RESTCONF and gNMI wire protocols. Any new emerging standards would gladly be added to FreeCONF.

Capable enough standards

YANG, RESTCONF and gNMI are all very capable. Networks have had years of experiences with these standards and their capabilities are well documented but you should judge for yourself of course.

Implementing the standard

Challenge: I need a library that implements the standard.
Answer: FreeCONF and others
Challenge: Awareness the standard exists:
Answer: You’re here aren’t you? Spread the word, share this site.
Question: Time to implement the standard:
Answer: Take it slow. See “What can it do” for strategies to get there in parallel to what you are doing now.

1.5 - What can it do?

Use cases on FreeCONF’s utility today.

It is important to remember FreeCONF is a software library. If you are looking for final applications to download see some options in Resources. With a library however you are in complete control of building what you need.

Ideal

While this is the goal:

Idea

The basic unit starts with this:

Basic Unit

With this basic unit, you can do a lot.

Just a few example use cases

Document existing configuration - Describe your current config in YANG and generate docs. You’ll have to make sure your application code and your YANG stay in sync but this is true of any manual documentation. This way you can separate content from format. YANGs ability to reuse definitions it very useful.
Validating configuration parser - Use FreeCONF to load your configuration files into your application. Follows the Inversion of Control pattern. Your configuration document will be accurate and configuration will be validated before it hits your code.
Edit live configuration - Loading configuration from files or REST is no different. You can expose just the configuration that you want to support live changing like debug log level or manage a list of users.
Decoupling metrics systems - Unlike configuration or structured logging, implementing metrics almost always requires coupling your application code to a proprietary metrics service locking you in. Use FreeCONF to decouple that. You don’t need to capture RPCs, config or events in YANG. See one of the metrics examples like Prometheus to see how it works.
Wrap existing REST APIs - Having a schema to a legacy API helps consumers and also makes the API available to RESTCONF and gNMI tools.
General purpose REST API - I may be biased, but RESTCONF APIs are far more powerful, easier to maintain and easier to consume then OpenAPIs. “Management” is really a perspective. Putting items in your web cart is really just managing objects in another object.
Admin Portal - While this may seem out of scope for FreeCONF, the ability to organize large sets of configuration quickly while providing both a REST API and dynamic user interfaces is quite useful. You can send configuration to endpoints by calling Ansible playbooks or editing Chef databags.
Bridge to other protocols - Just like RESTCONF and gNMI are just wire protocols to the basic unit, you can write a bridge to other protcols like JMX, SNMP, DOCSYS, COMI or proprietary protocols.
File Parser - Generate code based from YANG to decode bytes in a file or wire protocol like msgpack or protobuf that do not contain self-describing meta data.
Generate OpenAPI definitions - YANG is truly a superset of all other protocols. Partly because it is very complete but also because you can extend the YANG to include meta data used in other definition systems like defining 404 errors. Here is one project that converts definitions. But you can create your own fairly easily by converting the YANG to JSON using fc-yang utility and then use jinja or your favorite templing system to convert to openapi definitions.
Generate gRPC definitions - Rationale and strategy is same as above
Validate YANG files - Just use the YANG parser in FreeCONF to parse YANG files and look for specific things or ensure they are syntactically correct.
Generating end-to-end tests - Walk each item in the YANG and generate isolated tests for just that item.

This is not the end of the list by no means, just ideas.

1.6 - Project history

Brief history around why the project was created.

By Douglas Hubler

In 2001, I was first introduced to management software for a telephony system, which marked the beginning of my involvement in this field. Together with my colleague, Damian Kremenski, we developed a basic meta language to assist with management duties, which proved to be crucial in our work. Over the course of the next decade, we dedicated ourselves to developing plugin APIs to allow third-party developers to add services to the telephony system, with our modeling language serving as a key component.

While we were proud of the management system we had created, we were also aware of the challenges involved in accessing it due to its proprietary meta language and reliance on plugins.

In 2014, I transitioned to a new role in the networking field, where I became familiar with two management standards, NETCONF and YANG. YANG is a modeling language, while NETCONF is a protocol that leverages said management model. Given my experience in developing a management meta language previously, I quickly recognized the value of YANG. While NETCONF was powerful, I found its newer REST-based counterpart, RESTCONF, to be more exciting as it was expected to increase adoption rates.

As a user of these standards, I was impressed with how they addressed interoperability, which led me to envision the potential for an open-source implementation of these standards that could be used in general-purpose application development. This solution would break free from the limitations of proprietary management tools and the plugin-oriented systems that I had contributed to in the past.

Between 2015 and the present day, I have been actively implementing the YANG and RESTCONF standards. Throughout the development process, my main objectives were to ensure that the library did not interfere with developers’ work and to make complex tasks possible while keeping simpler tasks easy.

In 2023, ground broke on supporting Python. After many attempts to port FreeCONF to other languages, we finally found a way that is scalable, maintainable and familiar to Python developers. This same strategy can be used to support every language. This is still a work in progress but well past proof of concept.

Also in 2023 support was added gNMI; a wire protocol that can work in replace of RESTCONF. Not only did adding gNMI bring a greater set of DevOps tool support it remains completely abstract and optional for all applications that use FreeCONF.

I cannot see DevOps in the same way now. Ansible playbooks look like homework ripe with fleeting assumptions. Terraform plugins look endless. JuJu looks sexy but too few applications. Datadog integration looks like a decision that cannot be easily undone. All of these tools are amazing and necessary but if each added support for these management standards then they would remove the burdon put on their community to build elaborate bridges just to use them.

2 - Getting Started

5 minute guide to using FreeCONF today.

From an active Go project with go command in the PATH run this command

go get github.com/freeconf/restconf

Install latest Python package.

pip install https://github.com/freeconf/lang/releases/download/v0.1.0-alpha/freeconf-0.1.0-py3-none-any.whl
fc-lang-install -v

The fc-lang-install script should not need root and simply downloads a single file containing FreeCONF core to ~/.freeconf/bin.

Step 2.) Create a basic YANG file

Create file ./hello.yang to describe your application management.

module hello {
	leaf message {
		type string;
	}
}

Step 3.) Create an application

There is a lot of flexibility on how you initialize RESTCONF, here is one way.

Go
Python

Create file main.go

package main

import (
	"log"

	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/source"
)

// This is not specific to restconf, just example data structure for your
// application
type MyApp struct {
	Message string
}

func main() {
	// your app instance
	app := MyApp{}

	// where to find YANG files
	ypath := source.Any(source.Path("."), restconf.InternalYPath)

	// organize modules.
	d := device.New(ypath)

	// register your application module. you can register as many as you want here.
	//   param 1 - name of the module, "hello.yang" must exist in ypath
	//   param 2 - code that connects (bridges) from your App to yang interface
	//             there are many options in nodeutil package to base your
	//             implementation on.  Here we use reflection because our yang file aligns
	//             with out application data structure.
	rootNode := nodeutil.Reflect{}.Object(&app)
	d.Add("hello", rootNode)

	// select RESTCONF as management protocol. gNMI is option as well
	restconf.NewServer(d)

	// this will apply configuration and starting RESTCONF web server
	if err := d.ApplyStartupConfigFile("./startup.json"); err != nil {
		log.Fatal(err)
	}

	select {}
}

Create file getting-started.py

from freeconf import restconf, source, device, parser, node, source, nodeutil
from threading import Event

# Represents a basic python application. There are no requirements imposed
# by FreeCONF on how you develop your application
class MyApp:
    def __init__(self):
        self.message = None

app = MyApp()

# The remaining is FreeCONF specific and shows how to build a management interface
# to your python application

# specify all the places where you store YANG files
ypath = source.any(
    source.path("."),                   # director to your local *.yang files
    source.restconf_internal_ypath()    # required for restconf protocol support
)

# load and validate your YANG file(s)
mod = parser.load_module_file(ypath, "hello")

# device hosts one or more management "modules" into a single instance that you
# want to export in the management interface
dev = device.Device(ypath)

# connect your application to your management implementation.
# there are endless ways to to build your management interface from code generation,
# to reflection and any combination there of.  A lot more information in docs.
mgmt = nodeutil.Node(app)

# connect parsed YANG to your management implementation.  Browser is a powerful way
# to dynamically control your application can can be useful in unit tests or other contexts
# but here we construct it to serve our management API
b = node.Browser(mod, mgmt)

# register your app management browser in device.  Device can hold any number of browsers
dev.add_browser(b)

# select RESTCONF as management protocol. gNMI is option as well or any custom or 
# future protocols
s = restconf.Server(dev)

# this will apply configuration including starting the RESTCONF web server
dev.apply_startup_config_file("./startup.json")

# simple python trick to wait until ctrl-c shutdown
Event().wait()

Step 4.) Create a startup config

Create this file with static configuration for your application and accompanying RESTCONF server in the file startup.json

{
    "fc-restconf" : {
        "web" : {
            "port": ":8080"
        }
    },
    "hello" : {
        "message" : "hello"
    }
}

Step 5.) Startup your application

Go
Python

go run ./main.go

python3 ./getting-started.py

Step 6.) Test your API

# read data
curl http://localhost:8080/restconf/data/hello:

# set data
curl -X PATCH -d '{"message":"goodbye"}' http://localhost:8080/restconf/data/hello:

# read back new data
curl http://localhost:8080/restconf/data/hello:

Done

That’s it. See YANG Primer for more information on the YANG modeling langage and Basic Components on to integrate with your existing code base.

3 - Reference

Complete technical documentation in detail.

3.1 - YANG primer

Quickstart on the most used parts of the management modeling language.

YANG is a language utilized for modeling the management functions of applications. The primary function of YANG files are to enable remote tools to comprehend a server’s management functions, enable servers to accurately provide defined management functions, and generate documentation for individuals seeking to understand a server’s management functions. As a software engineer who implements management functions, it is crucial to understand how to create YANG files. Although there are numerous books on the subject, this document will provide a high-level understanding. It is recommended to consult the YANG RFC for a comprehensive specification of the language.

If you are familiar with other interface definition languages such as gRPC or OpenAPI, you will find several similarities with YANG. However, YANG extends beyond these languages to account for management-specific aspects.

It is essential to note that YANG is not limited to RESTCONF and can be employed with other communication protocols, such as NETCONF or custom protocols.

If you familiar with YANF and RESTCONF already, some these responses might look different. See Compliance on how to control this.

Data Definitions

module

Every YANG file starts module {} statement. All further definitions are contained inside the {} brackets.

module car {
  prefix "c";
  namespace "yourcompany.com";
  revision 2023-03-11;

  // all further definitions here
}

There is always just a single module in a YANG file.

RFC reference

leaf

module car {
  leaf driver {
    type string;
  }
}

If a server used this YANG file, then possible response for requesting data via curl might be:

curl https://server/restconf/data/car:

{
 "driver": "joe"
}

While the base URL of /restconf/data/ is not strictly neccessary, it is pretty standard for all RESTCONF servers.

RFC reference

all possible leaf types

Name	Description
binary	Any binary data
bits	A set of bits or flags
boolean	“true” or “false”
decimal64	64-bit signed decimal number
empty	A leaf that does not have any value
enumeration	One of an enumerated set of strings
identityref	A reference to an abstract identity
instance-identifier	A reference to a data tree node
int8	8-bit signed integer
int16	16-bit signed integer
int32	32-bit signed integer
int64	64-bit signed integer
leafref	A reference to a leaf instance
string	A character string
uint8	8-bit unsigned integer
uint16	16-bit unsigned integer
uint32	32-bit unsigned integer
uint64	64-bit unsigned integer
union	Choice of member types

RFC reference

container

module car {
  container engine {}
}

Possible request/responses:

curl https://server/restconf/data/car:

{
 "engine": {}
}

RFC reference

list

module car {
  list cylinders {
    leaf firingOrder {
      type int32;
    }
  }
}

Possible request/responses:

curl https://server/restconf/data/car:

{
  "cylinders":[
    {
      "firingOrder": 1
    }
    {
      "firingOrder": 4
    }
    {
      "firingOrder": 3
    }
    {
      "firingOrder": 2
    }
  ]
}

Most list will have a key when referencing a particular item in the list.

module car {
  list cylinders {
    key num;
    leaf num {
      type int32;
    }
    leaf firingOrder {
      type int32;
    }
  }
}

curl https://server/restconf/data/car:

{
  "cylinders":[
    {
      "num": 1,
      "firingOrder": 1
    }
    {
      "num": 2,
      "firingOrder": 4
    }
    {
      "num": 3,
      "firingOrder": 3
    }
    {
      "num": 4,
      "firingOrder": 2
    }
  ]
}

curl https://server/restconf/data/car:cylinders=1

{
  "num": 1,
  "firingOrder": 1
}

RFC reference

container/leaf

module car {
  leaf driver {
    type string;
  }
  container engine {
    leaf cylinders {
      type int32;
    }
  }
}

Possible request/responses:

curl https://server/restconf/data/car:

{
 "driver": "joe"
 "engine": {
  "cylinders": 6
 }
}

RFC reference

leaf-list

module car {
  leaf-list owners {
    type string;
  }
}

Possible request/responses:

curl https://server/restconf/data/car:

{
 "owners": ["joe","mary"]
}

leaf-lists can have all the same types as leaf, only it would container multiples of said type.

RFC reference

rpc

module car {
  rpc start {}
}

Possible request/responses:

curl -X POST https://server/restconf/data/car:start

# no response but car should start otherwise you'd get an error

RFC reference

rpc/input/output

module car {
  rpc drive {
    input {
      leaf throttle {
        type int32;
      }
    }
    output {
      leaf acceleration {
        type int32;
      }
    }
  }
}

Possible request/responses:

curl -X POST -d '{"throttle":32}' https://server/restconf/data/car:drive

{
  "acceleration": 30
}

RFC reference

container/action/input/output

For historical reasons, action is exactly like rpc except rpcs are only allowed inside module and action is used everywhere else.

module car {
  
  rpc drive {} // correct

  container engine {
    action start {} // correct
  }
}

module car {
  
  action drive {} // INCORRECT, only rpc here
  
  container engine {
    rpc start {} // INCORRECT, only action here
  }
}

RFC reference

notification

module car {
  notification flatTire {}   
}

Possible request/responses:

curl https://server/restconf/data/car:flatTire

data: {"notificaton":{"eventTime":"2013-12-21T00:01:00Z"}}

RFC reference

notification/leaf

module car {
  notification flatTire {
    leaf tireCount {
      type int32;
    }
  }   
}

Possible request/responses:

curl https://server/restconf/data/car:flatTire

data: {"notificaton":{"eventTime":"2013-12-21T00:01:00Z","event":{"tireCount":1}}}

RFC reference

anydata

module car {
  anydata glovebox;
}

Possible request/responses:

curl https://server/restconf/data/car:

{
  "glovebox" {
    "papers" : ["registration", "manual"],
    "napkinCount" : 30
  }
}

RFC reference

Organizational

group/uses

Grouping is simply a way to reuse a set of definitions

This YANG

module car {
  
  leaf driver {
    type string;
  }

  uses engineDef;

  grouping engineDef {
    container engine {
      leaf cylinders {
        type int32;
      }
    } 
  }
}

is equivalent to this YANG:

module car {
  
  leaf driver {
    type string;
  }

  container engine {
    leaf cylinders {
      type int32;
    }
  } 
}

RFC reference

choice/case

When you want to ensure there is just one of a number of definitions. If you are familiar with gRPC, this is like oneof. Some languages call this union:

module {
  choice nameDoesntMatter {
    leaf a {
      type string;
    }
    leaf b {
      type string;
    }
    leaf c {
      type string;
    }
  }
}

This means if a exists, then b and c cannot.

This you have multiple items in the option, you can wrap them with a case statement.

module {
  choice nameDoesntMatter {
    case nameAlsoDoesntMatter1 {
      leaf a {
        type string;
      }
      leaf aa {
        type string;
      }
    }
    leaf b {
      type string;
    }
    leaf c {
      type string;
    }
  }
}

This means if a and/or aa exist, then b and c cannot.

RFC reference

typedef

typedef are simply a way to reuse a leaf type

This YANG

module {
  leaf driver {
    type string;
  }
}

is equivalent to this YANG

module {
  
  typedef person {
    type string;
  }

  leaf driver {
    type person;
  }
}

This would be more useful then type has more definitions associated with it and more opportunities to reuse the typedef.

RFC reference

Metrics

Metrics are just definitions that are marked config false.

module {

  leaf speed {
    config false;  // <-- Metric HERE
    type int32;
  }

  leaf color {
    type string;
  }
}

module {

  container stats {
    config false;  // <-- children of a containers are metrics too
    leaf count {
      type int32;
    }
    leaf rate {
      type int32;
    }
  }
}

RFC reference

Constraints

There are a lot more types of contraints, but here are a few;

number types

module car {
  leaf cylinders {
    type int32 {
      range "1..12";
    }
  }
}

You can have any number of range items that you need.

RFC reference

string types

module car {
  leaf color {
    type string {
      pattern "[a-z]*"; // lowercase
    }
  }
}

You can have any number of pattern items that you need.

RFC reference

Extensions

You can customize YANG files with data that is specific to your application. Extensions will be ignored by any systems that do not support your customizations.

module car {
  prefix my;

  extension secret;

  leaf owner {
    type string;
    my:secret;
  }
}

extensions have have any number of arguments:

module car {
  prefix my;

  extension secret {
    argument "vault";
  }

  leaf owner {
    type string;
    my:secret "safe";
  }

  leaf history {
    type string;
    my:secret "jerry";
  }
}

RFC reference

Even this is not an exhaustive, but still more useful contructs:

import - pull in select YANG from another file. RFC reference
include - pull in all YANG from another file. RFC reference
default - value to use for leafs when no value is supplied. RFC reference
augment - used with grouping/uses to add definitions to a grouping. RFC reference
refine - also used with uses to alter specific definition including adding constraints. RFC reference
leafref - a reference to another leaf’s data that must exist. Kinda like a foreign key RFC reference
when - data definitions that only exist when certain data is true. RFC reference
must - a constraint that is tied to other leaf’s data. RFC reference
identity - system wide enum that can have heirarchies. RFC reference
feature - denote parts of the YANG that are only valid if a feeature it on. RFC reference
revision - track versions of your YANG. RFC reference
error-message - control the error messaging. RFC reference
ordered-by - control the order of lists. RFC reference

Example YANG files

toaster.yang - Used as the “hello world” or “TODOs” application for YANG. It demonstrates common YANG items.
fc-restconf.yang - If you use FreeCONF, this is the configuration of the RESTCONF service you’d be using to handle your RESTCONF interface.
ietf-inet-types.yang - IETF types are useful to import as common set of field types/

3.2 - Basic components

Six basic components when developing with FreeCONF.

When using the FreeCONF developer API you’ll likely be interfacing with these six FreeCONF components:

Module - the parsed YANG model (i.e. AST). From this single meta object, you can reach the meta definitions for every single item in the YANG file including custom extensions.
Node - custom code you write that connects the definitions in Module to your application. As your management API is called, Nodes are responsible for returning more Nodes to navigate your application. Nodes are also responsible for handling each configuration field, each metric, each notification and each function. You can use a variety of methods to implement your Nodes from:
- writing each by hand
- reflection
- generated from code
- library of base implementations you have developed
- nodes that read or write from/to JSON, YAML, XML
- nodes that read and write from/to a database
- nodes that proxy to other nodes including remote ones to create a client
- nodes that extend other nodes
- yet to be developed
- any combination of the above
Selection - a pairing of a node and a meta definition that lets you navigate and operate on your application’s live objects.
Browser - Where to get the first Selection object known as the root Selection.
Device - Holds a set of Browsers that you elect to make available in your management API.
Server - Handles the communication in and out of your Device. It understands the protocol and responsible for interoperability with clients communicating with your Device.

classDiagram
  Selection o--> Node
  Selection o--> Meta
  Device *-->  "1..*" Browser
  Server *--> Device
  Browser *--> Module
  Browser --> Selection : << creates >>
  Selection --> Selection : << creates >>
  Browser *--> Node
  Module *--> "1..*" Meta
  Node --> Node : << creates >>
  class Device{    
  }
  class Selection{
  }
  class Module{
  }
  class Meta{    
  }
  class Node{
  }
  class Server{    
  }

3.3 - Using RESTCONF

Interfacing with a RESTCONF API as a tool, CLI or script

This is about consuming a RESTCONF API, not building one. In short, if you know REST then you know RESTCONF. There are some really cool features however that will find useful and a few conventions that would be good to learn.

Adjustable data granularity

Traditional REST APIs struggle with what level of information to return for each GET request and this is called granularity. Return too little data and scripts will need to make constant trips for more. Return too much data and majority of it will likely be unused. Both situations cause delays and additional resource consumption.

RESTCONF APIs do not have this issue. Clients can precisely control the data they want from depth, to list pagination. From selecting fields to excluding fields.

With APIs developed with FreeCONF, each implemention has the control to only read the data that was requested. That is, granularity is not implemented as a reponse filters like GraphQL after the data has already been extracted.

Methods

No surprises here:

GET - Getting configuration, metrics and notifications. Notificatons are in SSE format detailed later in this document.
PATCH - Updating configuration. If objects are not found, they will be created so this is really an upsert.
PUT - Replacing configuration. If objects are found, they will removed first then recreated with the given configuration.
POST - Creating read-write data (e.g. config). If objects are not found, you will get an error.
DELETE - Deleting read-write data. If objects are not found, you will not get an error.
OPTIONS - Useful to test if user has access to certain data path

URL

Basic for is this : /restconf/data/{module}:{path}[?params]

module - name of the yang file So car.yang would be served at /restconf/data/car:
path - drill down into the objects. So to access the car tires, would be /restconf/data/car:tires. Only tricky part is drilling into lists. If you wanted to drill into front-left tire, you might use /restconf/data/car:tires=front-left. Drilling deeper still might be /restconf/data/car:tires=front-left/vendor
?params - For GET methods only, many params will help you limit the amount of data you return to save bandwidth and compute resources. More detailed information can be found in the specification but here is a quick summary:
- depth=N - limits the level of data returns to N levels in the hierarchy
- content=config - return only configuration data
- content=nonconfig - return only metric data
- with-defaults=trim - Do not return leaf values if they match the default value. This is useful for determining what a configuration user may have actually changed versus what configuration a device is actually using.
- fields=a;b/c - returns only select data paths. Note: you’ll need to encode parameters depending on your http client libraries. For example this would be fields=a%3db/c

Custom URL params

FreeCONF adds a few extra, useful parameters when retrieving data

fc.xfields=a;b/c - inverse of fields in that it returns all fields except specified. Again watch the encoding of the ; as detailed above.
fc.range=b/c!N-M - returns rows N thru M inclusive in list b/c
fc.max-node-count=N - increases or decreases the maximum allowed data to be returned. There are limits by default to ensure unbounded requests do not bog down system.

Examples

See specification for more details on how RESTCONF maps to REST.

Task	Method	Path	Description
Read	GET	`/restconf/data/car:`	Get’s all data (configuration and metrics) for car module
Read	GET	`/restconf/data/car:tire`	Get’s all data for all tires
Read	GET	`/restconf/data/car:tire=1`	Get’s all data for first car tire. Yes, seeing an equals in a URL can be disconcerting, but it is legal.
Update	PATCH	`/restconf/data/car:cruise` body:`{"desiredSpeed":65}`	Set cruise control desired speed
Read	GET	`/restconf/data/car:tire?c2-range=!1-2`	Get’s all data for car tires 1 and 2
Read	GET	`/restconf/data/car:tire?fields=wear%3did`	Get’s only tire id and wear level for all tires. `%3d` is encoded `=`.
Read	GET	`/restconf/data/car:tire?content=config&with-defaults=trim`	Get’s only configuration that is changed from the default for all tires
Create	POST	`/restconf/data/car:navigate` body:`{"destination":{"address":"10 Main st."}}`	Add a new destination address to navigation. This would only work if no naviation address was already set.
Delete	DELETE	`/restconf/data/car:navigate/destination`	Remove destination from navigation system.
RPC	POST	`/restconf/data/car:rotateTires`	Run a RPC to rotate the tires
RPC	POST	`/restconf/data/car:rotateTires` body:`{"order":"clockwise"}`	Run a RPC to rotate the tires in specific order
RPC	POST	`/restconf/data/car:rotateTires` body:`{"order":"clockwise"}` response:`{"charge":30.00}`	Run a RPC to rotate the tires in specific order and return the cost.
Event Stream	GET	`/restconf/data/car:` response: `{"status":"running"}` `{"status":"stopped","reason":"flat"}`	Stream of events regarding the car status
Event Stream	GET	`/restconf/data/car:?filter=status%3dstopped` response: `{"status":"stopped","reason":"flat"}`	Stream only events that cause car to stop. `%3d` is encoded `=`.

Events

RESTCONF delivers events using SSE(Server State Events) over HTTP. This is simply a stream per event stream. HTTP/2 allows for an unlimited number of streams over a single connection. Each event is serialized JSON followed by 2 end-of-line characters so you know the event message boundaries.

There is no special library required to read these messages and you can subscribe to as many event streams as you want w/o opening a new connection.

Subscribing to events in web browser:

// this looks like a new connection, but HTTP/2 sends it over existing connection
// to unsubscribe, call  events.close();
const events = new EventSource("/restconf/data/car:updates");
events.addEventListener("message", (e) => {
   console.log(e.data);
 });

Subscribing to events in CLI

$ curl https://server/restconf/data/car:updates
data: {"tire":{"wear":80}}

data: {"tire":{"wear":70}}

Examples subscription paths:

Path	Description
`updates`	Any changes to car
`updates?filter=tire/wear<20`	Any changes to car when the tire wear is less than 20

When using REST API to build a web interface, checkout model assisted web UI.

3.4 - Node Development

Using the FreeCONF API to build your management interface

Nodes are used to bridge your application to the YANG modeled management interface. As a software engineer you decide how those nodes are designed. Each management request constructs the node heirarchy to navigate to the part of the application to perform the management operation. When the management operation is complete, the nodes are garbage collected. Parallel requests construct different set of nodes so you can keep state in your request logic.

Unlike gRPC or OpenAPI, FreeCONF was designed to allow you to work with your existing application data structures and functions, not separate, generated ones. One reason for the difference is RESTCONF let’s the client pick and choose the data they want returned. Generating full data structures only to be partially populated is inefficient at best. Luckily FreeCONF gives you many options for how to implement your nodes, from a blank slate, to reflection, to code generation to anywhere in between.

Each node starts from a root node registered with the Browser object. Each node controls how its child nodes are constructed and therefore you can mix node implementations to suit each part of your existing application’s design.

If you’re not sure where to start, typically a YANG model uses the similar naming for fields and data structures of the application so I would start with Reflect and Extend. If your code becomes too repetative, then you can start to look at strategies for generating code to replace parts of your current code. This phased approach is useful because you’ll know what code to generate and replace.

3.4.1 - Interface

Understanding the role of a node.

You are unlikely to be implementing this interface as a developer but this gives a great understanding of all the responsiblities of a node. Each of the node base struct will give you options to control certain aspects of these functions.

type Node interface {

	// Child is called when navigating, creating or deleting a "container" or "list".
	// The request fields contain the nature of the request.
	//
	// Params:
	//  ChildRequest - contains the nature (e.g. get, delete, new) and details of the
	//                 request (e.g. identity name of the container or list )
	//
	// Return:
	//   Node - Return nil entity does not exist otherwise node implementation of the underlying
	//          node requested
	Child(r ChildRequest) (child Node, err error)

	// Next is called for items in a list when navigating, creating or deleting items in the
	// "list".
	//
	// Params:
	//  ListRequest - contains the nature (e.g. get, delete, new) and details of the
	//                 request (e.g. key, row number)
	//
	// Return:
	//   Node - Return nil entity does not exist otherwise node implementation of the underlying
	//          node requested
	//   []val.Value - If a key was defined in YANG, AND the request is for the next item
	//           in the list, then you must also return the key expressed in val.Value array
	Next(r ListRequest) (next Node, key []val.Value, err error)

	// Field is called to read or write "leaf" or "leaf-list" items on container/list items or
	// even root module.
	//
	// Params:
	//   FieldRequest - identity of the field and the contains the nature (e.g. read or write)
	//             tip: if the YANG defines the field as config:false, you can assume the field is
	//             only called for reading and you can ignore the flag for read v.s. write
	//
	//   *ValueHandle - contains a single the "Val" that with either:
	//                   a.) contain the value to be set or
	//                   b.) be expected to be set to the requested value to be read
	Field(r FieldRequest, hnd *ValueHandle) error

	// Choose is only called when there is a need to know which single case between several "choice/case"
	// sets are currently true.  This is only called when reading.
	//
	// Params:
	//   Selection - current selection that is handling the choice investigation
	//   *Choice - name for the choice (there could be several in a given container)
	//
	// Return:
	//   *ChoiceCase - which case is currently valid (there can be only one)
	//
	Choose(sel Selection, choice *meta.Choice) (m *meta.ChoiceCase, err error)

	// BeginEdit is called simply to inform a node when it is about to be edited including deleting or creating.
	// While there is nothing required of nodes to do anything on this call, implementations might have
	// very important, internal operations they need to perform like obtaining write locks for example. It is
	// also an opportunity to reject the edit for whatever reason by returning an error.
	//
	// It important to know this is called on **every** parent node for any edit to any of their children.
	// This is known as "event bubbling" and can be helpful when parents are in a better position to handle
	// operations for children.  You can easily distinguish this situation from the NodeRequest parameter.
	// See https://github.com/freeconf/restconf/wiki/Edit-Node-Traversal for details on when this is called
	// in particular in the order
	//
	// Params:
	//  NodeRequest - contains the nature (e.g. get, delete, new)
	BeginEdit(r NodeRequest) error

	// EndEdit is called simply to inform a node after it has been edited including deleting or creating.
	// While there is nothing required of nodes to do anything on this call, implementations might have
	// very important, internal operations they need to perform like release write locks for example. It is
	// also an opportunity to apply changes to live application or persist data to permanent storage.
	//
	// It important to know this is called on **every** parent node for any edit to any of their children.
	// This is known as "event bubbling" and can be helpful when parents are in a better position to handle
	// operations for children.  You can easily distinguish this situation from the NodeRequest parameter.
	// See https://github.com/freeconf/restconf/wiki/Edit-Node-Traversal for details on when this is called
	// in particular in the order
	//
	// Params:
	//  NodeRequest - contains the nature (e.g. get, delete, new)
	EndEdit(r NodeRequest) error

	// Action(rpc/action) is called when caller wished to run a 'action' or 'rpc' definition.  Input can
	// be found in request if an input is defined.  Output only has to be returned for
	// definitions that declare an output.
	//
	// Params:
	//  ActionRequest - In addition to the identity name of the action or rpc, it may contains the "input"
	//        to the action if one was defined in the YANG.
	//
	// Return:
	//  Node - If there is an expected response from action ("output" defined in YANG) then this would be
	//        the Node implementation of that data.
	Action(r ActionRequest) (output Node, err error)

	// Notify(notification) is called when caller wish to subscribe to events from a node.

	// Notificationsare unique with respect to resources are allocated that survive the original
	// request so implementation should try not keep references to any more resources than neccessary
	// to ensure they do not reference objects that become stale or that impose large chunks of memory
	// unknowningly and unneccesarily.
	//
	// Params:
	//   NotifyRequest - In addition to the identity name of the notification, this contains the stream
	//        to write events to.
	//
	// Return:
	//   NotifyCloser - Function to called to stop streaming events. This will be called for example when
	//      client closes the event stream socket either naturally or because of a network disconnect.
	//
	Notify(r NotifyRequest) (NotifyCloser, error)

	// Peek give an opportunity for API callers to gain access to the objects behind a node.
	//
	// Params:
	//   Selection - current selection that is handling the peeking
	//   interface{} - Who (or under what context) is attempting to peek. This can be used by
	//         node implementation to decide what is returned or if anything at all is returned.
	//         It may also be ingored.  This is up to
	//
	// Return:
	//   interface{} - Can be anything.
	//
	// This can be considered a violation of abstraction so implementation is not gauranteed and
	// can vary w/o any compiler warnings.  This can be useful in unit testing but uses outside this
	// should be used judicously.
	Peek(sel Selection, consumer interface{}) interface{}

	// Context provides an opportunity to add/change values in the request context that is passed to
	// operations on this node and operations to children of this node.  FreeCONF does not put or
	// require anything in the context, it is meant as a way to make application or request specific
	// data available to nodes.
	//
	// A popular use of context is to store the user and or user roles making the request so
	// operations can authorize or log user operations.  See RESTCONF request filters for one
	// way to implement that.
	Context(sel Selection) context.Context
}

3.4.2 - Edit lifecycle

In-depth details on developing your applications using FreeCONF and understanding the sequence of callbacks when editing node structures.

This is an in-depth, advanced description about how Node implementations handle edits.

When initiating an edit on your application you might want to call custom functions to create data structures at the beginning of the edit, at the end or at some point along the way. You might also want to implement custom locking. This document will go thru your options using FreeCONF library.

type Node interface {

    // Hook before an edit is made. opportunity to reject potential edit or create locks
    BeginEdit(r NodeRequest) error

    // Called to navigate and process edits
    Child(r ChildRequest) (child Node, err error)
    Next(r ListRequest) (next Node, key []val.Value, err error)
    Field(r FieldRequest, hnd *ValueHandle) error


    // Hook after an edit is made. opportunity to trigger persistance, finalize edit
    // or free locks
    EndEdit(r NodeRequest) error
}

Edit: Delete a node

[Root]
    [Parent]
        [Target]    <-- Delete Target
            [Child]
                [Child-Child]

Node	BeginEdit	End Edit
Root	3. EditRoot=false,New=false,Delete=false	6. EditRoot=false,New=false,Delete=false
Parent	2. EditRoot=false,New=false,Delete=false	5. EditRoot=false,New=false,Delete=false
Target	1. EditRoot=true,New=false,Delete=true	4. EditRoot=true,New=false,Delete=true

NOTES:

The [Child] and [Child-Child] nodes never get called when their parent is deleted
Every single ancester node of [Target] would be called on every edit no matter how deep the tree was. This would mean you could implement a single global write lock on a [Root] for any write edit if that is what you needed. Or a single write to database of entire tree on any edit.

Edit:Create a new node thru a target node

[Root]
    [Parent]
        [Target]    <-- New Child-Child request passed to Target
            [Child]
                [Child-Child]

Node	BeginEdit	End Edit
Root	3. EditRoot=false,New=false,Delete=false	10. EditRoot=false,New=false,Delete=false
Parent	2. EditRoot=false,New=false,Delete=false	9. EditRoot=false,New=false,Delete=false
Target	1. EditRoot=true,New=false,Delete=false	8. EditRoot=true,New=false,Delete=false
Child	4. EditRoot=true,New=false,Delete=false	7. EditRoot=true,New=false,Delete=false
Child-Child	5. EditRoot=true,New=true,Delete=false	6. EditRoot=true,New=true,Delete=false

Edit: Edit leaf on a node

[Root]
    [Parent]
        [Target]    <-- Edit leaf property on Target
            [Child]
                [Child-Child]

Node	BeginEdit	End Edit
Root	3. EditRoot=false,New=false,Delete=false	6. EditRoot=false,New=false,Delete=false
Parent	2. EditRoot=false,New=false,Delete=false	5. EditRoot=false,New=false,Delete=false
Target	1. EditRoot=true,New=false,Delete=false	4. EditRoot=true,New=false,Delete=false

3.5 - Resources

Links of useful resources on the Internet

FreeCONF plays an important part of a larger community of people bringing together specifications, information, products and projects. If you would like to submit a link, please submit a merge request.

FreeCONF

Information/Specifications

YANG/RESTCONF on wikipedia
RFCs
Network Programmability with YANG - Book by Benoît Claise on RESTCONF/YANG and related technologies.
gNMI is gRPC over HTTP/2 based protocol as an alternative to RESTCONF.

Examples

Industry YANG files - From yangcatalog.org project. Useful to see if others have modeled similar applications and/or examples models.
More Industry YANG files - Same as aboive for the OpenConfig quasi-standard
Bartender - Open source, robotic bartender built with FreeCONF.

Tools and implementations

gNMIc - a gNMI CLI client that provides full support for Capabilities, Get, Set and Subscribe RPCs with collector capabilities. Of particular interest is the terminal that supports tab complete on YANG.
Ultra Config - Manage configurations in the cloud.
MG Soft - Thick client to walk RESTCONF compatible endpoints
YumaWorks - C++ Drivers for RESTCONF among other things
Ansible - RESTCONF - Support for RESTCONF and related protocols with documentation here
Ansible - URI - RESTCONF client is so simple and Ansible support for REST APIs might be easier to use.
ygot - Alternative to FreeCONF for generating Go code from YANG
Watsen Networks SZTP Support - For bootstrapping startup configs even before call home requests.
Terraform via gNMI - Early experiment but could be used to start something more.
gNMI Gateway - Includes Prometheus exporter

Honorable Mention

FreeCONF doesn’t currently support these tools but they are in reach. If these protocol were added to FreeCONF or these projects added support for RESTCONF or gNMI.

NETCONF

NETCONF is a TCP/IP socket based protocol

Yuma123 - Open source implementation of NETCONF
SaltStack
Chef Support

SNMP

SNMP is still used for metrics today and might provide some value and would be a simple enough protocol to integrate with.

InfluxDB Support

3.6 - Compliance

Items of note regarding IETF RFC compliance

FreeCONF’s intentions is for strict compliance with IETF RFCs and OpenConfig to allow interoperability with other tools. In order to make the standards appeal to a more general audiance however, FreeCONF has a few extensions that detailed below. If you find a violation or concern, please bringing this to the attention of the project via an issue or discussion. Any ommisions in compliance are likely due to limited resources to implement and contributions would be welcomed.

3.6.1 - Extensions

Listing of optional extensions

By default FreeCONF’s intentions is for strict compliance with RFC. There are some minor additions that can often be disabled should you require only strict compliance.

Useful options

JSON w/o namespaces

If you submit application/yang-data+json in either Accept or Content-Type in HTTP request headers, you get JSON with namespaces per RFC

Strict Example Response:

{
    "car:engine":{"speed":10,"x:status":running}
}

With no MIME types or ?simplified in URL, you get

Optional Example Response:

{
    "engine":{"speed":10,"status":running}
}

PUT, PATCH or POST requests can have namespaces or not, doesn’t matter. But if they are supplied, they must be correct.

Rational(s):

namespaces expose how YANG files are organized and often that shouldn’t matter to API consumer. It might matter to another machine (m2m) in which case proper MIME types would be used and this simplified version wouldn’t matter.
added noise in data and primary data stutter
only useful for rare name collisions which should be avoided anyway to make APIs more clear anyway

Simplified base URL

If you submit application/yang-data+json in either Accept or Content-Type in HTTP request headers, these are the base URLS as per the RFC

Strict Base URLs

/restconf/data/{module:}... - CRUD and `actions`
/restconf/operations/{module:}...  - `rpc`
/restconf/streams/{module:}... - `notifications`

With no MIME types or ?simplified in URL, you get

Optional Base URL:

/restconf/data/{module:}... - CRUD, `rpcs`, `actions` and `notifications`

Rational(s)

Separation was likely for historical reasons and causes unnecessary complicates API usages

RPC Input/Output Wrapping

If you submit application/yang-data+json in either Accept or Content-Type in HTTP request headers, you get JSON with namespaces per RFC

Strict Example Input:

{
    "car:input": {
      "gas":30
    }
}

Strict Example Output:

{
    "car:output": {
      "cost":43.56
    }
}

With no MIME types or ?simplified in URL, you get

Optional Example Input:

{
    "gas":30
}

Optional Example Output:

{
    "cost":43.56
}

Rational:

input and output object wrappers add no value

Uploading files

Details

Rational:

Being able to upload files to a REST API is fundamental to any REST API.

URL Parameters

FreeCONF specific URL params

Rational:

Very useful when creating web UIs

Recursive YANG definitions

Referencing a grouping from inside the grouping is not allowed in YANG but is allowed in FreeCONF YANG parser

module fileStructure {
  
  grouping directory {
    leaf name {
      type string;
    }
    container parent {
      uses directory;
    }
  }
}

Rational:

Recursive relationships should be avoided when possible but sometimes unavoidable so being able to model it is important. Implementors should use caution to ensure the data model is not recursive as well. Also when developing tools that navigate the YANG AST, be sure to use flag that denotes a recursive defintion to avoid indefinite recusion.

3.6.2 - RFCs

Listing of IETF RFCs implemented

There might be more, but these are the high level, implemented RFCs

RFC 6020 - YANG 1.0
RFC 7950 - YANG 1.1
RFC 7951 - JSON encoding
RFC 8040 - RESTCONF - sans XML and etag support
RFC 8525 - YANG Library
RFC 8071 - RESTCONF Call Home

3.7 - Doc generation

Generating documentation from your yang files.

FreeCONF comes with a utility to help generate documentation. Your options include

HTML format
Markdown - Useful for storing in github.com, gitlab.com
Graphviz dot that can be then turned into SVG diagram that then can be added to top of HTML page
Raw JSON which can then be used as input to your own template script
Export template you can edit and pass back in

Usage

$ go run github.com/freeconf/yang/cmd/fc-yang doc -help

Usage of fc-yang:
  -f string
    	output format. available formats include html, md, json or dot. (default "none")
  -img-link string
    	Link to image for HTML templates. Default is (module-name).svg.
  -module string
    	Module to be documented.
  -off value
    	disable this feature.  You can specify -off multiple times to disable multiple features. You cannot specify both on and off however.
  -on value
    	enable this feature.  You can specify -on multiple times to enable multiple features. You cannot specify both on and off however.
  -t string
    	Use the template instead of the builtin template.
  -title string
    	Title. (default "RESTful API")
  -x	export the builting template to stdout. You can then edit template and pass it back in using -t option.  Be sure to pick correct format.
  -ypath string
    	Path to YANG files

Optional Graphviz

For the SVG, you will need to install Graphviz.

Example:

sudo apt install graphviz

Example commands

Generate HTML with SVG image at top

fc-yang doc -f dot -module fc-restconf -ypath yang > fc-restconf.dot
dot -Tsvg fc-restconf.dot -o fc-restconf.svg
fc-yang doc -f html -module fc-restconf -title "FreeCONF RESTCONF" -ypath yang > fc-restconf.html

Example output - HTML with SVG

fc-restconf API

Customize by tweaking the template

fc-yang doc -f html -module my-mod -x > doc.template
# Edit doc.template
fc-yang doc -f html -t doc.template -title "My API" -module my-mod -ypath yang > my-api.html

3.8 - Errors

Returning 401, 409 and 400 class errors

When coding your node.Node implementation, if you return an error from any function, your http clients will receive a 500 error, but if you want to return an error with a different HTTP status code, you can use or wrap one of the predefined errors

import (
  "github.com/freeconf/yang/fc"
)

  ...
  // Option #1 - straight 401 error
  return fc.UnauthorizedError 

  // Option #2 - enhanced but still an 401 error
  return fmt.Errorf("Bad ACL %w", fc.UnauthorizedError)

4 - Examples

Every type of example imaginable and a few not imaginable.

Most of these examples come from the FreeCONF examples git repo.

4.1 - Next Step

After “Getting Started”

This document goes past Getting Started by exploring YANG’s fundamental management types:

configuration
metrics
RPCs
notifications

We’ll demonstrate thru example by writing software that represents a car. This car has configuration such as speed, metrics such as the odometer, rpcs such as start and stopping the car and events such as getting a flat tire.

You can structure your code however you like, but here we are breaking this example into five separate files:

Car main source
Car unit test
YANG to describe management interface
Management source using FreeCONF
Management unit test

1. Car main source

This is just normal code to model the car. Notice there are no references to FreeCONF library, this is just code as you might normally write it.

Go
Python

file : car.go

package car

import (
	"container/list"
	"fmt"
	"math/rand"
	"time"
)

// ////////////////////////
// C A R - example application
//
// This has nothing to do with FreeCONF, just an example application written in Go.
// that models a running car that can get flat tires when tires are worn.
type Car struct {
	Tire []*Tire

	Miles   float64
	Running bool

	// When the tires were last rotated
	LastRotation int64

	// Default speed value is in yang model file and free's your code
	// from hardcoded values, even if they are only default values
	// units milliseconds/mile
	Speed int

	// How fast to apply speed. Default it 1s or "miles per second"
	PollInterval time.Duration

	// Listeners are common on manageable code.  Having said that, listeners
	// remain relevant to your application.  The manage.go file is responsible
	// for bridging the conversion from application to management api.
	listeners *list.List
}

// CarListener for receiving car update events
type CarListener func(UpdateEvent)

// car event types
type UpdateEvent int

const (
	CarStarted UpdateEvent = iota + 1
	CarStopped
	FlatTire
)

func (e UpdateEvent) String() string {
	strs := []string{
		"unknown",
		"carStarted",
		"carStopped",
		"flatTire",
	}
	if int(e) < len(strs) {
		return strs[e]
	}
	return "invalid"
}

func New() *Car {
	c := &Car{
		listeners:    list.New(),
		Speed:        1000,
		PollInterval: time.Second,
	}
	c.NewTires()
	return c
}

// Stop will take up to poll_interval seconds to come to a stop
func (c *Car) Stop() {
	c.Running = false
}

func (c *Car) Start() {
	if c.Running {
		return
	}
	go func() {
		c.Running = true
		c.updateListeners(CarStarted)
		defer func() {
			c.Running = false
			c.updateListeners(CarStopped)
		}()
		for c.Speed > 0 {
			poll := time.NewTicker(c.PollInterval)
			for c.Running {
				for range poll.C {
					c.Miles += float64(c.Speed)
					for _, t := range c.Tire {
						t.endureMileage(c.Speed)
						if t.Flat {
							c.updateListeners(FlatTire)
							return
						}
					}
				}
			}
		}
	}()
}

// OnUpdate to listen for car events like start, stop and flat tire
func (c *Car) OnUpdate(l CarListener) Subscription {
	return NewSubscription(c.listeners, c.listeners.PushBack(l))
}

func (c *Car) NewTires() {
	c.Tire = make([]*Tire, 4)
	c.LastRotation = int64(c.Miles)
	for pos := 0; pos < len(c.Tire); pos++ {
		c.Tire[pos] = &Tire{
			Pos:  pos,
			Wear: 100,
			Size: "H15",
		}
	}
}

func (c *Car) ReplaceTires() {
	for _, t := range c.Tire {
		t.Replace()
	}
	c.LastRotation = int64(c.Miles)
}

func (c *Car) RotateTires() {
	x := c.Tire[0]
	c.Tire[0] = c.Tire[1]
	c.Tire[1] = c.Tire[2]
	c.Tire[2] = c.Tire[3]
	c.Tire[3] = x
	for i, t := range c.Tire {
		t.Pos = i
	}
	c.LastRotation = int64(c.Miles)
}

func (c *Car) updateListeners(e UpdateEvent) {
	fmt.Printf("car %s\n", e)
	i := c.listeners.Front()
	for i != nil {
		i.Value.(CarListener)(e)
		i = i.Next()
	}
}

// T I R E
type Tire struct {
	Pos  int
	Size string
	Flat bool
	Wear float64
	Worn bool
}

func (t *Tire) Replace() {
	t.Wear = 100
	t.Flat = false
	t.Worn = false
}

func (t *Tire) checkIfFlat() {
	if !t.Flat {
		// emulate that the more wear a tire has, the more likely it will
		// get a flat, but there is always a chance.
		t.Flat = (t.Wear - (rand.Float64() * 10)) < 0
	}
}

func (t *Tire) endureMileage(speed int) {
	// Wear down [0.0 - 0.5] of each tire proportionally to the tire position
	t.Wear -= (float64(speed) / 100) * float64(t.Pos) * rand.Float64()
	t.checkIfFlat()
	t.checkForWear()
}

func (t *Tire) checkForWear() bool {
	return t.Wear < 20
}

///////////////////////
// U T I L

// Subscription is handle into a list.List that when closed
// will automatically remove item from list.  Useful for maintaining
// a set of listeners that can easily remove themselves.
type Subscription interface {
	Close() error
}

// NewSubscription is used by subscription managers to give a token
// to caller the can close to unsubscribe to events
func NewSubscription(l *list.List, e *list.Element) Subscription {
	return &listSubscription{l, e}
}

type listSubscription struct {
	l *list.List
	e *list.Element
}

// Close will unsubscribe to events.
func (sub *listSubscription) Close() error {
	sub.l.Remove(sub.e)
	return nil
}

file : car.py

import freeconf.nodeutil
import time
import threading
import random

# If these strings match YANG enum ids then they will be converted automatically.  Ints would
# work as well
EVENT_STARTED = "carStarted"
EVENT_STOPPED = "carStopped"
EVENT_FLAT_TIRE = "flatTire"


# Simple application, no connection to management 
# C A R
# Your application code.
#
# Notice there are no reference to FreeCONF in this file.  This means your
# code remains:
# - unit test-able
# - Not auto-generated from model files
# - free of annotations/tags
class Car():

    def __init__(self):
        
        # metrics/state
        self.miles = 0
        self.running = False
        self.thread = None

        # potential configuable fields
        self.speed = 1000
        self.new_tires()
        self.poll_interval = 1.0 #secs

        # Listeners are common on manageable code.  Having said that, listeners
        # remain relevant to your application.  The manage.go file is responsible
        # for bridging the conversion from application to management api.
        self.listeners = []

    def start(self):
        if self.running:
            return
        self.thread = threading.Thread(target=self.run, name="Car")
        self.thread.start()

    def stop(self):
        """
          Will take up to poll_interval seconds to come to a stop
        """
        self.running = False

    def run(self):
        try:
            self.running = True
            self.update_listeners(EVENT_STARTED)
            while self.running:
                time.sleep(self.poll_interval)
                self.miles = self.miles + self.speed
                for t in self.tire:
                    t.endure_mileage(self.speed)
                    if t.flat:
                        self.update_listeners(EVENT_FLAT_TIRE)
                        return
        finally:
            self.running = False
            self.update_listeners(EVENT_STOPPED)

    def on_update(self, listener):
        """
        listen for car events like start, stop and flat tire
        """
        self.listeners.append(listener)
        def closer():
            self.listeners.remove(listener)
        return closer

    def update_listeners(self, event):
        print(f"car {event}")
        for l in self.listeners:
            l(event)    

    def new_tires(self):
        self.tire = []
        for pos in range(4):
            self.tire.append(Tire(pos))
        self.last_rotation = self.miles

    def replace_tires(self):
        for t in self.tire:
            t.replace()
        self.last_rotation = int(self.miles)
        self.start()

    def rotate_tires(self):
        first = self.tire[0]
        self.tire[0] = self.tire[1]
        self.tire[1] = self.tire[2]
        self.tire[2] = self.tire[3]
        self.tire[3] = first
        for i in range(len(self.tire)):
            self.tire[i].pos = i
        self.last_rotation = int(self.miles)


class Tire:
    def __init__(self, pos):
        self.pos = pos
        self.wear = 100
        self.size = "H15"
        self.flat = False
        self.worn = False

    def replace(self):
        self.wear = 100
        self.flat = False
        self.worn = False

    def check_if_flat(self):
        if not self.flat:
            # emulate that the more wear a tire has, the more likely it will
            # get a flat, but there is always a chance.
            self.flat = (self.wear - (random.random() * 10)) < 0

    def endure_mileage(self, speed):
        # Wear down [0.0 - 0.5] of each tire proportionally to the tire position
        self.wear -= (float(speed) / 100) * float(self.pos) * random.random()
        self.check_if_flat()
        self.check_for_wear()    

    def check_for_wear(self):
        self.wear < 20

2. Car unit test

Shown here for completeness.

Go
Python

file : car_test.go

package car

import (
	"fmt"
	"testing"
	"time"

	"github.com/freeconf/yang/fc"
)

// Quick test of car's features using direct access to fields and methods
// again, nothing to do with FreeCONF.
func TestCar(t *testing.T) {
	c := New()
	c.PollInterval = time.Millisecond
	c.Speed = 1000

	events := make(chan UpdateEvent)
	unsub := c.OnUpdate(func(e UpdateEvent) {
		fmt.Printf("got event %s\n", e)
		events <- e
	})
	t.Log("waiting for car events...")
	c.Start()

	fc.AssertEqual(t, CarStarted, <-events)
	fc.AssertEqual(t, FlatTire, <-events)
	fc.AssertEqual(t, CarStopped, <-events)
	c.ReplaceTires()
	c.Start()

	fc.AssertEqual(t, CarStarted, <-events)
	unsub.Close()
	c.Stop()
}

file : test_car.py

#!/usr/bin/env python3
import unittest 
import queue
import car

class TestCar(unittest.TestCase):

    # Quick test of car's features using direct access to fields and methods
    def test_server(self):
        c = car.Car()
        c.poll_interval = 0.01
        c.speed = 1000

        events = queue.Queue()
        def on_update(e):
            print(f"got event {e}")
            events.put(e)
        unsub = c.on_update(on_update)
        print("waiting for car events...")
        c.start()
        
        self.assertEqual(car.EVENT_STARTED, events.get())
        self.assertEqual(car.EVENT_FLAT_TIRE, events.get())
        self.assertEqual(car.EVENT_STOPPED, events.get())
        c.replace_tires()
        c.start()

        self.assertEqual(car.EVENT_STARTED, events.get())
        unsub()

        c.replace_tires()

        c.stop()

        
if __name__ == '__main__':
    unittest.main()

3. YANG to describe management interface

Come up with YANG file for the desired management API for your car. If you need help understanding YANG files there are a lot of references on the internet including this YANG primer.

file : car.yang

// Every yang file has a single module (or sub-module) definition.  The name of the module
// must match the name of the file. So module definition for "car" would be in "car.yang".
// Only exception to this rule is advanced naming schemes that introduce version into 
// file name.
module car {

	description "Car goes beep beep";
	
	revision 2023-03-27;  // date YYYY-MM-DD is typical but you can use any scheme 

	// globally unique id when used with module name should this YANG file mingles with other systems
	namespace "freeconf.org"; 

	prefix "car"; // used when importing definitions from other files which we don't use here

    // While any order is fine, it does control the order of data returned in the management
	// interface or the order configuration is applied. You can mix order of metrics and 
	// config, children, rpcs, notifications as you see fit

	
	// begin car root config...

	leaf speed {
		description "how many miles the car travels in one poll interval";
	    type int32;
		units milesPerSecond;
		default 1000;
	}

	leaf pollInterval {
		description "time between traveling ${speed} miles";
		type int32;
		units millisecs;
		default 1000;
	}

	// begin car root metrics...

	leaf running {
		description "state of the car moving or not";
		type boolean;
		config false;
	}

	leaf miles {
		description "odometer - how many miles has car moved";
		config false;
	    type decimal64 {
			fraction-digits 2;
		}
	}

	leaf lastRotation {
		description "the odometer reading of the last tire rotation";
		type int64;
		config false;
	}

	// begin children objects of car...

	list tire {
		description "rubber circular part that makes contact with road";
		
		// lists are most helpful when you identify a field or fields that uniquely identifies
		// the items in the list. This is not strictly neccessary.
		key pos;

        leaf pos {
			description "numerical positions of 0 thru 3";
            type int32;
        }

		// begin tire config...

        leaf size {
			description "informational information of the size of the tire";
            type string;
            default "H15";
        }

        // begin tire metrics

		leaf worn {
			description "a somewhat subjective but deterministic value of the amount of
			  wear on a tire indicating tire should be replaced soon";
            config false;
            type boolean;
        }

        leaf wear {
			description "number representing the amount of wear and tear on the tire. 
			  The more wear on a tire the more likely it is to go flat.";
            config false;			
            type decimal64 {
				fraction-digits 2;
			}
        }

        leaf flat {
			description "tire has a flat and car would be kept from running. Use
			   replace tire or tires to get car running again";
            config false;
            type boolean;
        }

		// begin tire RPCs...

		action replace {
			description "replace just this tire";

			// simple rpc with no input or output.

			// Side note: you could have designed this as an rpc at the root level that
			// accepts tire position as a single argument but putting it here makes it
			// more natural and simple to call.
		}
	}

	// In YANG 'rpc' and 'action' are identical but for historical reasons you must only 
	// use 'rpc' only when on the root and 'action' when inside a container or list. 

	// begin car RPCs...

	rpc reset {
		description "reset the odometer"; // somewhat unrealistic of a real car odometer
	}

    rpc rotateTires {
        description "rotate tires for optimal wear";
    }

    rpc replaceTires {
        description "replace all tires with fresh tires and no wear";
    }

	rpc start {
		description "start the car if it is not already started";
	}

	rpc stop {
		description "stop the car if it is not already stopped";
	}

	// begin of car events...

    notification update {
        description "important state information about your car";

		leaf event {
			type enumeration {
				enum carStarted {

					// optional. by default the values start at 0 and increment 1 past the 
					// previous value.  Numbered values may be even ever be used in your programs
					// and therefore irrelevant. Here I define a value just to demonstrate I could.
					value 1; 

				}
				enum carStopped;
				enum flatTire;				
			}
		}
    }
}

4. Management source using FreeCONF

Here we use FreeCONF’s nodeutil.Node that uses reflection to implement the managment functions.

This is far from the only way to implement this, you can generate code or implement your own logic that implements the node.Node interface.

Go
Python

file : manage.go

package car

import (
	"reflect"
	"time"

	"github.com/freeconf/yang/meta"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

// ///////////////////////
// C A R    M A N A G E M E N T
//
// Manage your car application using FreeCONF library according to the car.yang
// model file.
//
// Manage is root handler from car.yang. i.e. module car { ... }
func Manage(car *Car) node.Node {

	// We're letting reflect do a lot of the work when the yang file matches
	// the field names and methods in the objects.  But we extend reflection
	// to add as much custom behavior as we want
	return &nodeutil.Node{

		// Initial object. Note: as the tree is traversed, new Node instances
		// will have different values in their Object reference
		Object: car,

		// implement RPCs
		OnAction: func(n *nodeutil.Node, r node.ActionRequest) (node.Node, error) {
			switch r.Meta.Ident() {
			case "reset":
				// here we implement custom handler for action just as an example
				// If there was a Reset() method on car the default OnAction
				// handler would be able to call Reset like all the other functions
				car.Miles = 0
			default:
				// all the actions like start, stop and rotateTire are called
				// thru reflecton here because their method names align with
				// the YANG.
				return n.DoAction(r)
			}
			return nil, nil
		},

		// implement yang notifications (which are really just event streams)
		OnNotify: func(p *nodeutil.Node, r node.NotifyRequest) (node.NotifyCloser, error) {
			switch r.Meta.Ident() {
			case "update":
				// can use an adhoc struct send event
				sub := car.OnUpdate(func(e UpdateEvent) {
					msg := struct {
						Event int
					}{
						Event: int(e),
					}
					// events are nodes too
					r.Send(nodeutil.ReflectChild(&msg))
				})

				// we return a close **function**, we are not actually closing here
				return sub.Close, nil
			}

			// there is no default implementation at this time, all notification streams
			// require custom handlers.
			return p.Notify(r)
		},

		// implement fields that are not automatically handled by reflection.
		OnRead: func(p *nodeutil.Node, r meta.Definition, t reflect.Type, v reflect.Value) (reflect.Value, error) {
			if l, ok := r.(meta.Leafable); ok {
				// use "units" in yang to tell what to convert.
				//
				// Other useful ways to intercept custom reflection reads:
				// 1.) incoming reflect.Type t
				// 2.) field name used in yang or some pattern of the name (suffix, prefix, regex)
				// 3.) yang extension
				// 4.) any combination of these
				if l.Units() == "millisecs" {
					return reflect.ValueOf(v.Int() / int64(time.Millisecond)), nil
				}
			}
			return v, nil
		},

		// Generally the reverse of what is handled in OnRead
		OnWrite: func(p *nodeutil.Node, r meta.Definition, t reflect.Type, v reflect.Value) (reflect.Value, error) {
			if l, ok := r.(meta.Leafable); ok {
				if l.Units() == "millisecs" {
					d := time.Duration(v.Int()) * time.Millisecond
					return reflect.ValueOf(d), nil
				}
			}
			return v, nil
		},
	}
}

file : manage.py

from freeconf import nodeutil, val

#
# C A R    M A N A G E M E N T
#  Bridge from model to car app.
#
# manage is root handler from car.yang. i.e. module car { ... }
def manage(c):

    # implement RPCs (action or rpc)
    def action(n, req):
        if req.meta.ident == 'reset':            
            c.miles = 0
            return None
        return n.do_action(req)
    
    # implement yang notifications which are really just events
    def notify(n, r):
        if r.meta.ident == 'update':
            def listener(event):
			    # events are nodes too
                r.send(nodeutil.Node(object={
                    "event": event
                }))
            closer = c.on_update(listener)
            return closer
        
        return n.do_notify(r)
    
    # implement fields that are not automatically handled by reflection.
    def read(_n, meta, v):
        if meta.units == 'millisecs':
            return int(v * 1000)
        return v

    def write(_n, meta, v):
        if meta.units == 'millisecs':
            return float(v) / 1000 # ms to secs
        return v

	# Extend and Reflect form a powerful combination, we're letting reflect do a lot of the CRUD
	# when the yang file matches the field names.  But we extend reflection
	# to add as much custom behavior as we want
    return nodeutil.Node(c,
        on_action = action, on_notify=notify, on_read=read, on_write=write)

5. Management unit test

Write unit test for you management API without starting a server. You can read or write configuration, call functions, listen to events or read metrics all from your unit test.

Go
Python

file : manage_test.go

package car

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

// Test the car management logic in manage.go
func TestManage(t *testing.T) {

	// setup
	ypath := source.Path("../yang")
	mod := parser.RequireModule(ypath, "car")
	app := New()

	// no web server needed, just your app and management function.
	brwsr := node.NewBrowser(mod, Manage(app))
	root := brwsr.Root()

	// read all config
	currCfg, err := nodeutil.WriteJSON(sel(root.Find("?content=config")))
	fc.AssertEqual(t, nil, err)
	expected := `{"speed":1000,"pollInterval":1000,"tire":[{"pos":0,"size":"H15"},{"pos":1,"size":"H15"},{"pos":2,"size":"H15"},{"pos":3,"size":"H15"}]}`
	fc.AssertEqual(t, expected, currCfg)

	// access car and verify w/API
	fc.AssertEqual(t, false, app.Running)

	// setup event listener, verify events later
	events := make(chan string)
	unsub, err := sel(root.Find("update")).Notifications(func(n node.Notification) {
		event, _ := nodeutil.WriteJSON(n.Event)
		events <- event
	})
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, 1, app.listeners.Len())

	// write config starts car
	n, err := nodeutil.ReadJSON(`{"speed":2000}`)
	fc.AssertEqual(t, nil, err)
	err = root.UpdateFrom(n)
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, 2000, app.Speed)

	// start car
	fc.AssertEqual(t, nil, justErr(sel(root.Find("start")).Action(nil)))

	// should be first event
	fc.AssertEqual(t, `{"event":"carStarted"}`, <-events)
	fc.AssertEqual(t, true, app.Running)

	// unsubscribe
	unsub()
	fc.AssertEqual(t, 0, app.listeners.Len())

	// hit all the RPCs
	fc.AssertEqual(t, nil, justErr(sel(root.Find("rotateTires")).Action(nil)))
	fc.AssertEqual(t, nil, justErr(sel(root.Find("replaceTires")).Action(nil)))
	fc.AssertEqual(t, nil, justErr(sel(root.Find("reset")).Action(nil)))
	fc.AssertEqual(t, nil, justErr(sel(root.Find("tire=0/replace")).Action(nil)))
}

func sel(s *node.Selection, err error) *node.Selection {
	if err != nil {
		panic(err)
	}
	return s
}

func justErr(_ *node.Selection, err error) error {
	return err
}

file : test_manage.py

#!/usr/bin/env python3
import unittest 
from freeconf import parser, node, source, nodeutil
import manage
import queue
import car
import time

# Test the car management API in manage.py
class TestManage(unittest.TestCase):

    def test_manage(self):
        # where YANG files are
        ypath = source.path("../yang")
        mod = parser.load_module_file(ypath, 'car')

        # create your empty app instance
        app = car.Car()

        # no web server needed, just your app instance and management entry point.
        brwsr = node.Browser(mod, manage.manage(app))

        # get a selection into the management API to begin interaction
        root = brwsr.root()

        # TEST CONFIG GET: read all the config
        curr_cfg = nodeutil.json_write_str(root.find("?content=config"))
        expected = '{"speed":1000,"pollInterval":1000,"tire":[{"pos":0,"size":"H15"},{"pos":1,"size":"H15"},{"pos":2,"size":"H15"},{"pos":3,"size":"H15"}]}'
        self.assertEqual(expected, curr_cfg)

        # verify car starts not running.  Here we are checking from the car instance itsel
        self.assertEqual(False, app.running)

        # SETUP EVENT TEST: here we add a listener to receive events and stick them in a thread-safe
        # queue for assertion checking later
        events = queue.Queue()
        def on_update(n):
            msg = nodeutil.json_write_str(n.event)
            events.put(msg)
        unsub = root.find("update").notification(on_update)

        # TEST CONFIG SET: write config
        root.update_from(nodeutil.json_read_str('{"speed":2000}'))
        self.assertEqual(2000, app.speed)

        # TEST RPC: start car
        root.find("start").action()

        # TEST EVENT: verify first event. will block until event comes in
        print("waiting for event...")
        self.assertEqual('{"event":"carStarted"}', events.get())
        print("event receieved.")

        # TEST EVENT UNSUBSCRIBE: done listening for events
        unsub()

        # TEST RPCS: just hit all the RPCs for code coverage.  You could easily add the
        # underlying car object is changed accordingly
        root.find("rotateTires").action()
        root.find("replaceTires").action()
        root.find("reset").action()
        self.assertEqual(0, app.miles)
        root.find("tire=0/replace").action()
        print("done")

if __name__ == '__main__':
    unittest.main()

What’s Next?

YANG primer - to learn more about building your YANG
Code generation - To assist with reducing the code to write and improving the compile-time safety between code and YANG file
More examples for building your management interface using Basic, Extends and Reflect

4.2 - Development Guide

Building Go nodes by using abstract class

Use cases:

high-level routing areas
list nodes
areas with not a lot of CRUD
bridges to systems that are not Go structs (e.g. DB, YAML, external REST APIs, etc.)
part of code generation

Highlevel routing

Code

Go
Python

package demo

type App struct {
	users  *UserService
	fonts  *FontManager
	bagels *BagelMaker
}

func NewApp() *App {
	return &App{
		users:  &UserService{},
		fonts:  &FontManager{},
		bagels: &BagelMaker{},
	}
}

type UserService struct{}
type FontManager struct{}
type BagelMaker struct{}


class App:

    def __init__(self):
        self.users = {}
        self.fonts = {}
        self.bagels = {}

YANG

module my-app {
    namespace "freeconf.org";
    prefix "a";

    container users {
        // ... more stuff here
    }
    container fonts {
        // ... more stuff here
    }
    container bagels {
        // ... more stuff here
    }
}

…then your node code can be this.

Go
Python

package demo

import (
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

func manage(a *App) node.Node {
	return &nodeutil.Basic{
		OnChild: func(r node.ChildRequest) (node.Node, error) {
			switch r.Meta.Ident() {
			case "users":
				return nodeutil.ReflectChild(a.users), nil
			case "fonts":
				return nodeutil.ReflectChild(a.fonts), nil
			case "bagels":
				return nodeutil.ReflectChild(a.bagels), nil
			}
			return nil, nil
		},
	}
}

You cannot use Reflect here because fields are private.

from freeconf import nodeutil

def manage_app(app):

    def child(req):
        if req.meta.ident == 'users':
            return nodeutil.Node(app.users)
        elif req.meta.ident == 'fonts':
            return nodeutil.Node(app.fonts)
        elif req.meta.ident == 'bagels':
            return nodeutil.Node(app.bagels)
        return None
    
    # while this could easily be nodeutil.Node, we illustrate a Basic
    # node should you want essentially an abstract class that stubs all 
    # this calls with reasonable default handlers
    return nodeutil.Basic(on_child=child)

Additional Files

Go
Python

file: manage_test.go

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

func TestManage(t *testing.T) {
	a := NewApp()
	ypath := source.Dir(".")
	m := parser.RequireModule(ypath, "my-app")
	bwsr := node.NewBrowser(m, manage(a))

	root := bwsr.Root()
	defer root.Release()
	actual, err := nodeutil.WriteJSON(root)
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, `{"users":{},"fonts":{},"bagels":{}}`, actual)
}

file: test_manage.py

#!/usr/bin/env python3
import unittest 
from freeconf import parser, nodeutil, node, source
from manage import manage_app
from app import App 

class TestManage(unittest.TestCase):

    def test_manage(self):
        app = App()
        ypath = source.path("..")
        m = parser.load_module_file(ypath, 'my-app')
        mgmt = manage_app(app)
        bwsr = node.Browser(m, mgmt)
        root = bwsr.root()
        try:
            actual = nodeutil.json_write_str(root)
            self.assertEqual('{"users":{},"fonts":{},"bagels":{}}', actual)
        finally:
            root.release()

if __name__ == '__main__':
    unittest.main()

4.3 - RESTCONF client

Connecting to a RESTCONF server

You can communicate with a RESTCONF server using a basic HTTP library. For simple applications this is a reasonable approach. If you want to “level-up” your API access to be able to walk the schema (i.e. YANG) then you can use the RESTCONF client in FreeCONF.

package demo

import (
	"fmt"
	"strings"
	"testing"

	"github.com/freeconf/examples/car"
	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/client"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/source"
)

func connectClient() {

	// YANG: just need YANG file ietf-yang-library.yang, not the yang of remote system as that will
	// be downloaded as needed
	ypath := restconf.InternalYPath

	// Connect
	proto := client.ProtocolHandler(ypath)
	dev, err := proto("http://localhost:9998/restconf")
	if err != nil {
		panic(err)
	}

	// Get a browser to walk server's management API for car
	car, err := dev.Browser("car")
	if err != nil {
		panic(err)
	}
	root := car.Root()
	defer root.Release()

	// Example of config: I feel the need, the need for speed
	// bad config is rejected in client before it is sent to server
	n, err := nodeutil.ReadJSON(`{"speed":100}`)
	if err != nil {
		panic(err)
	}
	err = root.UpsertFrom(n)
	if err != nil {
		panic(err)
	}

	// Example of metrics: Get all metrics as JSON
	sel, err := root.Find("?content=nonconfig")
	if err != nil {
		panic(err)
	}
	defer sel.Release()
	metrics, err := nodeutil.WriteJSON(sel)
	if err != nil {
		panic(err)
	}
	if metrics == "" {
		panic("no metrics")
	}

	// Example of RPC: Reset odometer
	sel, err = root.Find("reset")
	if err != nil {
		panic(err)
	}
	defer sel.Release()
	if _, err = sel.Action(nil); err != nil {
		panic(err)
	}

	// Example of notification: Car has an important update
	sel, err = root.Find("update")
	if err != nil {
		panic(err)
	}
	defer sel.Release()
	unsub, err := sel.Notifications(func(n node.Notification) {
		msg, err := nodeutil.WriteJSON(n.Event)
		if err != nil {
			panic(err)
		}
		fmt.Println(msg)
	})
	if err != nil {
		panic(err)
	}
	defer unsub()

	// Example of multiple modules: This is the FreeCONF server module
	rcServer, err := dev.Browser("fc-restconf")
	if err != nil {
		panic(err)
	}

	// Example of config: Enable debug logging on FreeCONF's remote RESTCONF server
	serverSel := rcServer.Root()
	defer serverSel.Release()
	n, err = nodeutil.ReadJSON(`{"debug":true}`)
	if err != nil {
		panic(err)
	}
	serverSel.UpsertFrom(n)
	if err != nil {
		panic(err)
	}
}

func TestClient(t *testing.T) {

	// setup -  start a server
	ypath := source.Path("../yang")
	serverYPath := source.Any(ypath, restconf.InternalYPath)
	carServer := car.New()
	local := device.New(serverYPath)
	local.Add("car", car.Manage(carServer))
	s := restconf.NewServer(local)
	defer s.Close()
	cfg := `{
		"fc-restconf": {
			"debug": true,
			"web" : {
				"port": ":9998"
			}
		},
		"car" : {
			"speed": 5
		}
	}`
	fc.RequireEqual(t, nil, local.ApplyStartupConfig(strings.NewReader(cfg)))

	connectClient()
}

4.4 - gNMI server

Adding gNMI server-side support to your application

gNMI is a alternative communication protocol to RESTCONF by openconfig.net](https://www.openconfig.net). You might use gNMI over RESTCONF because the services you want to use work with gNMI. With FreeCONF you can easily enable RESTCONF, gNMI or both at the same time.

How to add gNMI server support to your application

file: main.go

package main

import (
	"flag"
	"log"

	"github.com/freeconf/examples/car"
	"github.com/freeconf/gnmi"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/source"
)

// Connect everything together into a server to start up
func main() {
	flag.Parse()

	// Your app here
	app := car.New()

	// where the yang files are stored
	ypath := source.Any(gnmi.InternalYPath, source.Path("../yang"))

	// Device is just a container for browsers.  Needs to know where YANG files are stored
	d := device.New(ypath)

	// Device can hold multiple modules, here we are only adding one
	if err := d.Add("car", car.Manage(app)); err != nil {
		panic(err)
	}

	// Select wire-protocol gNMI to serve the device
	gnmi.NewServer(d)

	// apply start-up config normally stored in a config file on disk
	if err := d.ApplyStartupConfigFile("./startup.json"); err != nil {
		panic(err)
	}

	if !*testMode {
		// wait for ctrl-c
		log.Printf("server started")
		select {}
	}
}

var testMode = flag.Bool("test", false, "do not run in background (i.e. driven by unit test)")

To run this example

git clone https://github.com/freeconf/examples fc-examples
cd ./gnmi-server
go run .

Checkout FreeCONF gNMIc examples for interacting with this running service.

To add gNMI support to your Go application

Just add the Go dependency and setup for gnmi.NewServer where ever makes sense to your application.

go get github.com/freeconf/gnmi

4.5 - gNMIc

Using the gNMIc client to connect to your application

gNMIc is a utility that can connect to any appliction that supports gNMI protocol including applications. See how to add gNMI support to your application using the FreeCONF. gnmic is unique because you can use it from the terminal to diagnose issues or use it in production to send metrics to InfluxDB, Prometheus, Kafka, NATS.

Here are some working examples of using gnmic from terminal to connect to the car example application with gnmi server support added.

Setup

Downloading FreeCONF example source code

git clone https://github.com/freeconf/examples fc-examples

Setup gnmi server

Run the example gNMI server with car application registered with server

cd ./gnmi-server
go run .

Running gnmic

First Install the gNMIc client executable. Then, in a separate terminal, go to the directory containing gnmic commands.

cd openconfig-gnmi

Example commands

#!/usr/bin/env bash

set -euf -o pipefail

# Get Data Object
gnmic --config car.yml get --path car:/

# Get Data Value
gnmic --config car.yml get --path car:/speed

# Get Data : use model.
gnmic --config car.yml get --model car --path /


# Set Data Object
gnmic --config car.yml set --update  car:/:::json:::'{"speed":300}'

# Set Data Value
gnmic --config car.yml set --update-path  car:/speed --update-value 300


# Subscribe
timeout 5s \
  gnmic --config car.yml sub --model car --path / --sample-interval 1s --heartbeat-interval 2s || true

# Subscribe to just tire metrics : use model
timeout 5s \
  gnmic --config car.yml sub --mode once --model car --path /tire || true

# Subscribe to just tire metrics
timeout 5s \
  gnmic --config car.yml sub --mode once --path car:/tire || true

# Subscribe to just tire 1 metrics
timeout 5s \
  gnmic --config car.yml sub --mode once --path car:/tire=1 || true

Example Output

Command: gnmic --config car.yml --model car --path /

[
  {
    "source": "localhost:8090",
    "timestamp": 1680564190640829839,
    "time": "2023-04-03T19:23:10.640829839-04:00",
    "updates": [
      {
        "Path": "",
        "values": {
          "": {
            "lastRotation": 0,
            "miles": 4100,
            "running": true,
            "speed": 10,
            "tire": [
              {
                "flat": false,
                "pos": 0,
                "size": "15",
                "wear": 100,
                "worn": false
              },
              {
                "flat": false,
                "pos": 1,
                "size": "15",
                "wear": 79.28663660297573,
                "worn": false
              },
              {
                "flat": false,
                "pos": 2,
                "size": "15",
                "wear": 59.94265362733487,
                "worn": false
              },
              {
                "flat": false,
                "pos": 3,
                "size": "15",
                "wear": 38.2541625477213,
                "worn": false
              }
            ]
          }
        }
      }
    ]
  }
]

Tab Complete

Called “prompt” mode in gnmic.

gNMIc prompt

4.6 - Ansible

Configure any RESTCONF client from ansible

Ansible can be used to get, set and call functions in RESTCONF. You can use the Ansible RESTCONF module or you can use the Ansible uri module to make RESTCONF calls as I’ve done here.

Setup

Downloading FreeCONF example source code

git clone https://github.com/freeconf/examples fc-examples

Setup ansible

cd ansible
python3 -m venv venv
source ./venv/bin/activate
python -m pip install -r requirements.txt

file: requirements.txt

ansible==7.0.0
ansible-core==2.15.8

Running

Inventory file

all:
  hosts:
    car:
      http_port: 8090
      ansible_host: localhost

Get Config

Command: ansible-playbook -i hosts.yml get-config.yml

File: get-config.yml

---
- name: get configure
  hosts: car
  connection: httpapi
  gather_facts: False

  tasks:
  - name: get current speed
    uri:
      url: "http://{{ ansible_host }}:{{ http_port }}/restconf/data/car:?depth=1"
    register: results

  - debug:
      var: results.json

Example Output:

****
TASK [get current speed] *******************************************************
ok: [car]

TASK [debug] *******************************************************************
ok: [car] => {
    "results.json": {
        "lastRotation": 13100,
        "miles": 20900,
        "running": false,
        "speed": 300
    }
}

PLAY RECAP *********************************************************************
car                        : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Set Config

Command: ansible-playbook -i hosts.yml set-config.yml

File: set-config.yml

---
- name: configure
  hosts: car
  connection: httpapi
  gather_facts: False

  tasks:
  - name: change speed
    uri:
      url: "http://{{ ansible_host }}:{{ http_port }}/restconf/data/car:"
      method: PATCH
      body_format: json
      body: |
        {
            "speed":10
        }

Run RPCs

Here we run the replaceTires RPC

---
- name: rpc
  hosts: car
  connection: httpapi
  gather_facts: False

  tasks:
  - name: replace tires
    uri:
      url: "http://{{ ansible_host }}:{{ http_port }}/restconf/data/car:replaceTires"
      method: POST

4.7 - InfluxDB

Sending metrics to InfluxDB

InfluxDB Car Tire Wear

Demonstrates:

How to use export FreeCONF application metrics to InfluxDB w/o coupling your application code from InfluxDB

Details

The fcinflux.Service walks thru all the local management interfaces and auto-discovers all metrics (i.e. config false in YANG) and sends them to InfluxDB at a configurable frequency.

architecture Car Tire Wear

Inside the code

“Walking” the metrics.

FreeCONF API to recursively send metrics to InfluxDB.

package fcinflux

import (
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/val"
)

// nodeWtr implements the FreeCONF node interface and acts like a "writer"
// when traversing a node tree with Selection.UpsertInto(<thisWriter>). There
// are other ways to accomplish walking a tree but this seemed easiest.
func nodeWtr(s sink, m Metric) node.Node {
	fields := make(map[string]interface{})
	return &nodeutil.Basic{
		OnField: func(r node.FieldRequest, h *node.ValueHandle) error {
			fields[r.Meta.Ident()] = h.Val.Value()
			return nil
		},
		OnChild: func(r node.ChildRequest) (node.Node, error) {
			return nodeWtr(s, m), nil
		},
		OnNext: func(r node.ListRequest) (node.Node, []val.Value, error) {
			return nodeWtr(s, m), r.Key, nil
		},
		OnEndEdit: func(r node.NodeRequest) error {
			if len(fields) > 0 {
				m.Name = r.Selection.Path.String()
				return s.send(r.Selection.Context, m, fields)
			}
			return nil
		},
	}
}

Running the example

Setting up InfluxDB

docker run -d --name=influxdb -p 8086:8086 influxdb:2.7

Configure and grab API token

Open a web browser to Influx DB admin portal at http://localhost:8086.

possible architecture

Created and admin account, organization and initial database.

Start Up Car Application

cd fcinflux/cmd
API_TOKEN=<api token from clipboard> go run .

tip: If you lost your API Token or ready to use a token for production, just log into Influx and generate a new one in menu.

Graph data

Log into InfluxDB interface and graph the data.

possible architecture

Conclusion

Comparison to gNMIc approach

gnmic

This has advantage of working with all gNMIc compliant devices.

In general, FreeCONF is a library to build your own solutions, gNMIc is a utility to use in production as is. You can decide at any point which approach works for you without having to change your application code, just your deployment strategy.

Using this example code

If you wanted to use the approach here as is, you could import fcinflux into your application directly by calling go get github.com/freeconf/example but this example code may change without notice. It’s intention is to give you a starter project to customize as needed.

Expanding on this example code

Open a discussion about what you’d like to see or feel free to make an annoucement on what you built.

Ideal architecture:

possible architecture

4.8 - Prometheus

Sending metrics to Prometheus

You have a few options for integrating your metrics with Prometheus. You can use gNMIc or freeconf/example/fcprom module. This document explores the later.

Prometheus Car Tire Wear

Demonstrates:

How to use export FreeCONF application metrics to Prometheus w/o coupling your application code to Prometheus
How to use YANG extensions to improve to Prometheus results

Details

The fcprom.Bridge walks thru all the local management interfaces and auto-discovers all metrics (i.e. config false in YANG) and makes them available to Prometheus.

architecture Car Tire Wear

Because YANG doesn’t understand Prometheus’ metrics types like gauge or counter or know how you want to flatten metrics in YANG lists, the fcprom module uses YANG extensions to help you control the translation. YANG extensions are ignored by other systems.

Defining some extensions

module metrics-extension {
    prefix "m";
    namespace "freeconf.org";
    description "Classify metrics into certain types so tools like Prometheus can handle them correctly";
    revision 0000-00-00;
    
	extension gauge {
        description "a value that goes up and down.  this is the default type if not specfied";
	}

	extension counter {
        description "a value that only goes up and is always positive.  Roll over is normal as well as reset on restart";
	}

    extension multivariate {
        description "one or more fields (from same node) to use values as label in metric report";
    }
}

Using our extensions

module car {
	description "Car goes beep beep";

	revision 2023-03-27;
	namespace "freeconf.org";
	prefix "car";

    import metrics-extension {
        prefix "metric";
    }

	leaf running {
		type boolean;
		config false;
	}
	
	leaf speed {
		description "How fast the car goes";
	    type int32;
		units milesPerSecond;
		default 1000;
		metric:counter;
	}

	leaf miles {
		description "How many miles has car moved";
		config false;
	    type decimal64 {
			fraction-digits 2;
		}
	}

	leaf lastRotation {
		type int64;
		config false;
	}

	list tire {
		description "Rubber circular part that makes contact with road";
		key "pos";

		// used to help flatten metrics in lists
		metric:multivariate;

        leaf pos {
            type int32;
        }
        leaf size {
            type string;
            default 15;
        }
        leaf worn {
            config false;
            type boolean;
        }
        leaf wear {
            config false;
            type decimal64 {
				fraction-digits 2;
			}
        }
        leaf flat {
            config false;
            type boolean;
        }

		action replace {
			description "replace just this tire";
		}
	}

    container engine {        
        anydata specs;
    }

	rpc reset {
		description "Reset the odometer";
	}

    rpc rotateTires {
        description "Rotate tires for optimal wear";
    }

    rpc replaceTires {
        description "Replace all tires";
    }

    notification update {
        description "Important state information about your car";
		leaf event {
			type enumeration {
				enum carStarted {
					value 1;
				}
				enum carStopped;
				enum flatTire;				
			}
		}
    }
}

Running the example

Downloading FreeCONF example source code

git clone https://github.com/freeconf/examples fc-examples

Setup and Run Prometheues

1.) Download and install Prometheus

2.) Start Prometheus with the example configuration here.

cd fcprom
prometheus --config.file=prometheus.yml

file: prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:8090']

Running Application

cd fcprom/cmd
go run .

Render Graph

Go to http://localhost:9090/ and enter car_tire_wear as expression

Using `fcprom` in your applications

package main

import (
	"flag"
	"log"
	"strings"

	"github.com/freeconf/examples/car"
	"github.com/freeconf/examples/fcprom"
	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/source"
)

// Connect everything together into a server to start up
func main() {
	flag.Parse()

	// Your app here
	app := car.New()

	// where the yang files are stored
	ypath := source.Path("../../yang:..")

	// Device is just a container for browsers.  Needs to know where YANG files are stored
	d := device.New(ypath)

	// Device can hold multiple modules, here we are only adding one
	if err := d.Add("car", car.Manage(app)); err != nil {
		panic(err)
	}

	// Prometheus will look at all local modules which will be car and fc-restconf
	// unless configured to ignore the module
	p := fcprom.NewBridge(d)
	if err := d.Add("fc-prom", fcprom.Manage(p)); err != nil {
		panic(err)
	}

	// Select wire-protocol RESTCONF to serve the device.
	restconf.NewServer(d)

	// apply start-up config normally stored in a config file on disk
	config := `{
		"fc-restconf":{
			"web":{
				"port":":8090"
			}
		},
		"fc-prom" : {
			"service" : {
				"useLocalServer" : true
			}
		},
        "car":{"speed":10}
	}`
	if err := d.ApplyStartupConfig(strings.NewReader(config)); err != nil {
		panic(err)
	}

	if !*testMode {
		// wait for ctrl-c
		log.Printf("server started")
		select {}
	}
}

var testMode = flag.Bool("test", false, "do not run in background (i.e. driven by unit test)")

Conclusion

Comparison to gNMIc approach

gnmic

This has advantage of working with all gNMIc compliant devices.

Using this example code

If you wanted to use the approach here as is, you could import fcprom into your application directly by calling go get github.com/freeconf/example but this example code may change without notice. It’s intention is to give you a starter project to customize as needed.

Expanding on this example code

Open a discussion about what you’d like to see or feel free to make an annoucement on what you built.

Ideal architecture:

possible architecture

Tips for extending:

Consider adding more YANG extensions like metrics:ignore to skip data or metrics:label to override the default label.

4.9 - Slack

Send notifications to Slack

Car Updates

Demonstrates

How to send YANG notifications to Slack

Details

The fcslack.Service will selectively subscribe to a list of paths in the local application and send the event payload to a designated channel in Slack.

architecture Car Tire Wear

If any of the paths are invalid, you will get an error. This is good. Unlike scraping logs, if you don’t have your regex correct, you silently miss the event.

This example can be easily extended to send events to Kafka, AWS SQS, Pager Duty, etc.

Running the example

Downloading FreeCONF example source code

git clone https://github.com/freeconf/examples fc-examples

Setup Slack workspace, application and bot token

You will have to setup a Slack Application, register the application with your Slack workspace and generate a bot token.

Create an environment file with your settings

file : fcslack/cmd/env.sh


# Should start with xoxb- which means it is a "Bot token"
export SLACK_API_TOKEN=xoxb-5555555555555-5555555555555-5xyZ5xyZ5xyZ5xyZ5xyZ5xyZ

# Any channel you want, you can control each message if you want in your application
export SLACK_CHANNEL=testing-integration

Run car application with your settings

cd fcslack/cmd
source ./env.sh
go run .

You should see the {"event":"carStarted"} event right away in your slack channel, then in minute or so see the {"event":"flatTire"} and the {"event":"carStopped"} events.

Conclusion

FreeCONF is a library to build your own solutions. This code can work with any models so once you build it once you can use it everywhere. Also, you can have multiple systems subscribing to the same event streams so maybe you start with slack and then weave in PagerDuty integration once you’re confident this is a reliable event stream.

Using this example code

If you wanted to use the approach here as is, you could import fcslack into your application directly by calling go get github.com/freeconf/example but this example code may change without notice. It’s intention is to give you a starter project to customize as needed.

Ideas of Expanding

Have a yang extension that will require fcslack to subscribe to ensure that event is always sent.

main.go

package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/freeconf/examples/car"
	"github.com/freeconf/restconf"

	"github.com/freeconf/restconf/device"

	"github.com/freeconf/yang/source"

	"github.com/freeconf/examples/fcslack"
)

func main() {
	flag.Parse()

	// Your app here
	app := car.New()

	// where the yang files are stored
	ypath := source.Path("../../yang:../../car:..")

	// Device is just a container for browsers.  Needs to know where YANG files are stored
	d := device.New(ypath)

	// Device can hold multiple modules, here we are only adding one
	if err := d.Add("car", car.Manage(app)); err != nil {
		panic(err)
	}

	s := fcslack.NewService(d)

	if err := d.Add("fc-slack", fcslack.Manage(s)); err != nil {
		panic(err)
	}

	restconf.NewServer(d)

	// apply start-up config normally stored in a config file on disk
	config := fmt.Sprintf(`{
		"fc-restconf":{
			"debug": true,
			"web":{
				"port":":8090"
			}
		},
		"fc-slack" : {
			"client" : {
				"apiToken": "%s"
			},
			"subscription": [
				{
					"channel" : "%s",
					"module": "car",
					"path": "update"
				}
			]
		},
        "car":{"speed":100}
	}`, os.Getenv("SLACK_API_TOKEN"), os.Getenv("SLACK_CHANNEL"))

	// bootstrap config for all local modules
	if err := d.ApplyStartupConfig(strings.NewReader(config)); err != nil {
		panic(err)
	}

	if !*testMode {
		// wait for ctrl-c
		log.Printf("server started")
		select {}
	}
}

var testMode = flag.Bool("test", false, "do not run in background (i.e. driven by unit test)")

4.10 - Node/Reflect

When your application objects and field names mostly align with your YANG model

Use cases:

CRUD on Go structs
CRUD on Go maps or slices

Special notes

You don’t need perfect alignment with Go field names and YANG to use Reflect. Let Reflect do the heavy lifting for you and capture small variations by combining with Extend. To that end, do not expect magical powers from Reflect to coerse your custom field types to YANG types.
Currently Reflect doesn’t attempt to use reflection to implement notifications or actions/rpcs but again, you can combine Reflect with Extend.
Names in YANG can be camelCase, kabob-case or snake_case interchangablely. Your Go public field are obviously in CamelCase.

Simple example

When you happen to have perfect alignment of field names to names in YANG.

If you have application code like this…

package demo

type Contacts struct {
	Me            *User
	Users         []*User
	Size          int
	HarmlessValue string
}

type User struct {
	FullName string
}

…and YANG like this…

module contacts {
    
    namespace "freeconf.org";
    prefix "a";

    container me {
        // full-name, full_name or fullName all work
        leaf full-name {
            type string;
        }
    }
    list users {
        key full-name;
        // could be grouping/uses here, doesn't matter
        leaf full-name {
            type string;
        }
    }
    leaf size {
        type int32;
    }
}

…then your node code can be this.

package demo

import (
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

func manage(app *Contacts) node.Node {
	return &nodeutil.Node{Object: app}
}

…and you test like this.

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

func TestManageContact(t *testing.T) {
	app := &Contacts{}
	ypath := source.Dir(".")
	m := parser.RequireModule(ypath, "contacts")
	b := node.NewBrowser(m, manage(app))

	actual, err := nodeutil.WriteJSON(b.Root())
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, `{"size":0}`, actual)
}

Map example

Reflect also supports Go’s map interface. While this Go code’s lack of data structures that might make this difficult to use in Go, if you don’t need to handle this data in Go, then this is completely acceptable. Remember, you can add constraints to yang to ensure the data is validated properly.

If you have application code like this…

package demo

type JunkDrawer struct {
	Info map[string]interface{}
}

|| …and YANG like this…

module junk-drawer {
    container info {
        leaf name {
            type string;
        }
        container more {
            leaf size {
                type decimal64;
            }
        }
        list stuff {
            key anything;
            leaf anything {
                type enumeration {
                    enum one;
                    enum two;
                }
            }
        }
    }
}

…then your node code can be this.

package demo

import (
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

func manageJunkDrawer(app *JunkDrawer) node.Node {
	return &nodeutil.Node{Object: app}
}

…and you test like this.

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

func TestManageJunk(t *testing.T) {
	app := &JunkDrawer{Info: make(map[string]interface{})}
	ypath := source.Dir(".")
	m := parser.RequireModule(ypath, "junk-drawer")
	b := node.NewBrowser(m, manageJunkDrawer(app))

	actual, err := nodeutil.WriteJSON(b.Root())
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, `{"info":{}}`, actual)
}

tips:

useful for building validated RESTCONF APIs quickly to be filled in later with structs
good for handling a bulk set of configs

Field coersion

Reflect can do a decent job converting Go primatives to and from YANG leaf types: strings to numbers, numbers to enums, enums to strings, number types to other number types, etc.. If a conversion represents a loss of data or a type can cannot be convert safely, then an error is returned. To handle the conversion of these values yourself, you can use Extend or Reflect.OnField . Reflect.OnField is more suited over Extend when have a lot of fields of the same type that you want to reuse in a lot of places and not one-offs.

If you have application code like this…

package demo

import "time"

type Timely struct {
	LastModified time.Time
}

…and YANG like this…

module timely {
    namespace "freeconf.org";
    prefix "t";
    
    leaf lastModified {        
        type int64; // unix seconds
    }
}

…then your node code can be this.

package demo

import (
	"reflect"
	"time"

	"github.com/freeconf/yang/meta"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

func manageTimely(t *Timely) node.Node {
	timeType := reflect.TypeOf(&time.Time{})
	return &nodeutil.Node{
		Object: t,
		OnRead: func(ref *nodeutil.Node, m meta.Definition, t reflect.Type, v reflect.Value) (reflect.Value, error) {
			switch t {
			case timeType:
				at := v.Interface().(*time.Time)
				return reflect.ValueOf(at.Unix()), nil
			}
			return v, nil
		},
		OnWrite: func(ref *nodeutil.Node, m meta.Definition, t reflect.Type, v reflect.Value) (reflect.Value, error) {
			switch t {
			case timeType:
				return reflect.ValueOf(time.Unix(v.Int(), 0)), nil
			}
			return v, nil
		},
	}
}

Adhoc structs

Create an anonymous struct or a just a map. Useful for handing RPC requests or responses. Here we use it to create whole app.

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
)

// Load a YANG module from a string!
var mstr = `
module mystery {
	namespace "freeconf.org";
	prefix "m";

	leaf name {
		type string;
	}
}
`

func TestMystery(t *testing.T) {

	m, err := parser.LoadModuleFromString(nil, mstr)
	fc.AssertEqual(t, nil, err)

	t.Run("struct", func(t *testing.T) {

		// create a node from an adhoc struct
		app := struct {
			Name string
		}{
			"john",
		}
		manage := &nodeutil.Node{Object: &app}

		// verify works
		b := node.NewBrowser(m, manage)
		actual, err := nodeutil.WriteJSON(b.Root())
		fc.AssertEqual(t, nil, err)
		fc.AssertEqual(t, `{"name":"john"}`, actual)
	})

	t.Run("map", func(t *testing.T) {

		// create a node from map
		app := map[string]interface{}{"name": "mary"}
		manage := &nodeutil.Node{Object: app}

		// verify works
		b := node.NewBrowser(m, manage)
		actual, err := nodeutil.WriteJSON(b.Root())
		fc.AssertEqual(t, nil, err)
		fc.AssertEqual(t, `{"name":"mary"}`, actual)
	})
}

4.11 - Node/Extend

When you want to make adjustments to an existing node’s implementation

Use cases:

Few isolated changes to an existing node
Wrap a CRUD node but customize editing operations, actions or notifications

Special notes

If [Reflect]({ { < relref “reflect” > } } ) was a cake and [Basic]({ { < relref “basic” > } } ) was the plate under the cake, then Extend would be the frosting.
Extend is exactly like Basic but let’s you delegate anything to another node. So most of Basic's documentation also applies here.

`Reflect` with one exception

Go
Python

package demo

import "fmt"

type Bird struct {
	Name string
	X    int
	Y    int
}

func (b *Bird) GetCoordinates() string {
	return fmt.Sprintf("%d,%d", b.X, b.Y)
}


class Bird():

    def __init__(self, name, x, y):
        self.name = name
        self.x = x
        self.y = y

    def coordinates(self):
        return f'{self.x},{self.y}'

module bird {
	namespace "freeconf.org";
	prefix "b";

	leaf name {
		type string;
	}

	leaf location {
		config false;
		type string {
			pattern "[0-9.]+,[0-9.]+";
		}
	}
}

Go
Python

package demo

import (
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/val"
)

func manage(b *Bird) node.Node {
	return &nodeutil.Extend{
		Base: nodeutil.ReflectChild(b), // handles Name
		OnField: func(p node.Node, r node.FieldRequest, hnd *node.ValueHandle) error {
			switch r.Meta.Ident() {
			case "location":
				hnd.Val = val.String(b.GetCoordinates())
			default:
				// delegates to ReflectChild for name
				return p.Field(r, hnd)
			}
			return nil
		},
	}
}

from freeconf import nodeutil, val

def manage_bird(bird):
    base = nodeutil.Node(bird)
    
    def on_field(parent, req, write_val):
        if req.meta.ident == "location":
            return val.Val(bird.coordinates())
        return parent.field(req, write_val)
    
    return nodeutil.Extend(base, on_field=on_field)

Addition Files

Go
Python

file: manage_test.go

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

func TestManage(t *testing.T) {
	b := &Bird{Name: "sparrow", X: 99, Y: 1000}
	ypath := source.Dir(".")
	m := parser.RequireModule(ypath, "bird")
	bwsr := node.NewBrowser(m, manage(b))

	root := bwsr.Root()
	defer root.Release()
	actual, err := nodeutil.WriteJSON(root)
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, `{"name":"sparrow","location":"99,1000"}`, actual)
}

file: test_manage.py

#!/usr/bin/env python3
import unittest 
from freeconf import parser, nodeutil, source, node
from manage import manage_bird
from bird import Bird 

class TestManage(unittest.TestCase):

    def test_manage(self):
        app = Bird("sparrow", 99, 1000)
        ypath = source.path('..')
        m = parser.load_module_file(ypath, 'bird')
        mgmt = manage_bird(app)
        bwsr = node.Browser(m, mgmt)
        root = bwsr.root()
        try:
            actual = nodeutil.json_write_str(root)
            self.assertEqual('{"name":"sparrow","location":"99,1000"}', actual)
        finally:
            root.release()

if __name__ == '__main__':
    unittest.main()

4.12 - Node/anydata

YANG anydata can be used send a variety of values

YANG has a type called anydata which can be anything. Reflect’s default behavior is to keep this as whatever the source node sends. For RESTCONF web requests this:

map[string]interface{} - When given loose data
decimal64 - when a number
string - when given a string
io.Reader - when given a file uploaded thru form mime type. See Forms

4.13 - Code generation

Writing code that writes code

While code generation is available now, this is a placeholder for the documentation that is coming soon.

In the meantime you can browse some of the early code here.

4.14 - Debugging

Techniques for debugging

Debug Logging

import (
  "github.com/freeconf/yang/fc"
)

...
   // turn on debug logging
   fc.DebugLog(true)

Logging `node` activity

Use cases:

See all operations performed on a node when you’re not sure you are getting the right data in or out.

Usage

  // wrap all node activity to the node app recursively
  n := nodeutil.Dump(manageApp(app), os.Stdout)

Example Edit

BeginEdit, new=false, src=car
->speed=int32(10)
EndEdit, new=false, src=car

Example Read

<-speed=int32(10)
<-miles=decimal64(310.000000)

tips:

Take a look at the source and create your own dumper that is maybe better.
You can do this at root node or any part of the tree you want to inspect

4.15 - Unit Testing

Strategies unit testing management nodes

You don’t need any special utilities to unit test your management, just a few useful techniques. If you look thru a lot of the node implementations you’ll see a lot of unit tests exibiting how to test nodes.

Testing without full application

As your application, management and YANG file grow, loading the full application each time just to test a piece might become cumbersome. We can use a feature of YANG to import from another YANG file and FreeCONF’s ability to make this easy to test just our unit.

package demo

import (
	"testing"

	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
)

type Here struct {
	Penny int
}

func TestUnitTestingPartialYang(t *testing.T) {

	// where car.yang is stored
	ypath := source.Dir(".")

	// Define new YANG module on the fly that references the application
	// YANG file but we pull in just what we want
	m, err := parser.LoadModuleFromString(ypath, `
		module x {
			import deep {
				prefix "d";
			}

			// pull in just the piece we are interested in. Here it is
			// just a single penny
			uses d:here;
		}
	`)
	if err != nil {
		t.Fatal(err)
	}

	// We create a "browser" to just a unit of our application
	h := &Here{Penny: 1}
	manage := node.NewBrowser(m, nodeutil.ReflectChild(h))

	// Example : test getting config
	actual, err := nodeutil.WriteJSON(manage.Root())
	fc.AssertEqual(t, nil, err)
	fc.AssertEqual(t, `{"penny":1}`, actual)
}

4.16 - Web UIs

Building a Web UI interface to a RESTCONF implementation is pure joy. This is a few tips that will serve you well.

While automation is primarily about APIs, building administration portals enable non-developers to interact with your application and another point of access for developers.

Car Admin Portal

Notes:

This example web code is very basic, but you can use advanced web frameworks
Read how to use RESTCONF API to understand how to navigate data.
Serving your web interface with your application ensures they are always deployed together and therefore compatible.

Car Demo

// Basic calls to interact with the "car" RESTCONF API

let car = null;

// Reading config AND metrics together for the whole car
function update() {
    fetch("/restconf/data/car:").then((resp) => {
        return resp.json();
    }).then((data) => {
        car = data;
        let e = document.getElementById("car");
        e.innerHTML = `
            <div>Miles: ${data.miles} miles</div>
            <div>Running: ${data.running}</div>
            <div>Speed: ${data.speed} mps</div>
            ${data.tire.map((tire, index) =>
                `<div>Tire ${index}: Flat: ${tire.flat}, Wear: ${tire.wear}</div>`
            ).join('')}
        `;
    });
};

// Call one of the RPCs. ours are simple, no input or expected output but
// you could easily add them here
function run(action) {
    fetch(`/restconf/data/car:${action}`, {
        method: 'POST',
    }).then(() => {
        update();
    });
}       

// Update config edit.  
//   PATCH is like an upsert. 
//   PUT is like a delete and replace with this.
//   POST is create new
function speed(s) {
    fetch('/restconf/data/car:', {
        method: 'PATCH',
        body : JSON.stringify({
                speed: car.speed + s
            })
    }).then(() => {
        update();
    });                
}

// Subscribe to notifications. EventSource is part of every browser and works 
// on HTTP (i.e. http) but more efficient on HTTP/2 (i.e. https)
function subscribeToUpdateEventStream() {
    // this will appear as a pending request.
    const events = new EventSource("/restconf/data/car:update?simplified");
	events.addEventListener("message", e => {
        // this decodes SSE events for you to give you just the messages
        const log = document.getElementById("log");
        log.appendChild(document.createTextNode(e.data + '\n'));
	});
    // to unsubscribe just close the stream.
    //  events.close();
}

// Metrics change too often for subscription so just get the latest
// every 2s.
const pollInterval = 2000;
function pollforUpdates()  {
    update();
    setTimeout(pollforUpdates, pollInterval);
}

// initial read of data
pollforUpdates();
// watch for update events as they happen, no polling here
subscribeToUpdateEventStream();

<html><!DOCTYPE html>
<html lang="en">
    <head>
        <title>Car Demo</title>
        <meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, user-scalable=yes">
    </head>
    <body>  
        <div>
            <button onclick='speed(10)'>Faster</button>
            <button onclick='speed(-10)'>Slower</button>
            <button onclick='run("replaceTires")'>Replace Tires</button>
            <button onclick='run("rotateTires")'>Rotate Tires</button>
            <button onclick='run("reset")'>Reset Odometer</button>    
        </div>
        <div id="car"></div>
        <p><b>tip:</b> keep rotating the tires to see how long how many miles you can get without getting a flat.</p>
        <script src="/ui/app.js"></script>
        <h3>Log</h3>
        <pre id="log"></pre>
    </body>
</html>

Additional Notes:

Register custom request endpoints - Not everything has to be handled thru RESTCONF. Use standard web request handlers for any custom endpoints.
Use notification for interactive UIs - Notifications aren’t just for alerts. One of the more useful notifications is for data has changed in back-end from possibly another user edit and front-end should reload data
Consider a web-only module - You can serve any number of modules in an application should you need to isolate your web-only functions. For example car module and car-web module both from the same server.
Generate REST API Docs - Generate REST API docs to know what is available from REST.

#!/usr/bin/env sh
set -eu

FC_YANG="go run github.com/freeconf/yang/cmd/fc-yang"
YPATH=../car
MODULE=car

# HTML API Docs
${FC_YANG} doc -f dot -module ${MODULE} -ypath ${YPATH} > ${MODULE}.dot
dot -Tsvg ${MODULE}.dot -o ${MODULE}.svg
${FC_YANG} doc -f html -module ${MODULE} -title "Car REST API" -ypath ${YPATH} > ${MODULE}-api.html

# Markdown API Docs
${FC_YANG} doc -f md -module ${MODULE} -ypath ${YPATH} > ${MODULE}-api.md

File Uploading

File Upload

module file-upload {
    namespace "freeconf.org";
    prefix "f";

    rpc bookReport {
        input {
            leaf bookName {
                description "Name of the book";
                type string;
            }

            anydata pdf {
                description "PDF of file upload";
            }
        }
        output {
            leaf fileSize {
                description "";
                type uint64;
            }
        }
    }
}

Frontend

<html><!DOCTYPE html>
<html lang="en">
    <head>
        <title>Book Report Upload</title>
        <meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, user-scalable=yes">
    </head>

    <body>
        <table>
            <tr>
                <td><label for="bookName">Book Name:</label></td>
                <td><input id="bookName"></td>
            </tr>
        
            <tr>
                <td><label for="pdf">Upload:</label></td>
                <td><input type="file" id="pdf" onchange='upload()'></td>
            </tr>
        </table>
        <script src="/ui/file-upload.js"></script>
    </body>
</html>


// grab the form data and upload the file to RESTCONF rpc
function upload() {
    const bookName = document.getElementById("bookName").value;
    const pdf = document.getElementById("pdf").files[0];
    const form = new FormData();
    form.append("bookName", bookName); // YANG: leaf bookName { type string; }
    form.append("pdf", pdf);           // YANG: anydata pdf;
    fetch("/restconf/data/file-upload:bookReport", {
      method: "POST",
      body: form
    })
    .then(resp => resp.json())
    .then((data) => {
        window.alert(`success. file size is ${data.fileSize}`);
    });
}

Backend

Go will see the anydata as an io.Reader.

package main

import (
	"io"

	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
)

// matches rpc input of uploading a book report
type upload struct {
	BookName string
	Pdf      io.Reader
}

// handle just one thing, file upload of book report
func manageUploader() node.Node {
	return &nodeutil.Basic{
		OnAction: func(r node.ActionRequest) (node.Node, error) {
			switch r.Meta.Ident() {
			case "bookReport":

				// you should do more validation of input data
				var req upload
				r.Input.UpdateInto(nodeutil.ReflectChild(&req))
				data, err := io.ReadAll(req.Pdf)
				if err != nil {
					return nil, err
				}

				// we throw away the report, but you'd probably want to keep it

				// tell user the file size as proof we got it
				resp := struct {
					FileSize uint64
				}{
					FileSize: uint64(len(data)),
				}

				return nodeutil.ReflectChild(&resp), nil
			}
			return nil, nil
		},
	}
}

Model Driven UI

Being able to read the information in the YANG file from your web application is nothing short of game changing. Here are just a few of the possibilities:

client-side form validation
build forms dynamically including simple things like select list options from leaf enumerations
form labels from leaf names and tooltips from descriptions
list of available columns in a table
reports

Combine this with ability to extend the YANG with your own meta data the possibilites are endless. For example:

mark leafs as password fields
marking fields that require web custom handlers
fields that should be shown to advanced users
fields that should only show if feature flag is on

The path to the meta definitions is just /restconf/schema/{module}/ and requires header Accept: application/json to return the YANG file in JSON form. You can use all normal RESTCONF navigation features to drill in to the just the part of the YANG file you are interested in.

File Upload

<html><!DOCTYPE html>
<html lang="en">
    <head>
        <title>Car Model</title>
        <meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, user-scalable=yes">
    </head>
    <body>
        <h2>Edit Car</h2>
        <div id="form"></div>
        <script src="/ui/model-driven.js"></script>
    </body>
</html>

// load the schema (YANG) and the object for editing
function load(module) {
    const parent = document.getElementById("form");
    Promise.allSettled([
        fetch(`/restconf/data/${module}:`)
            .then(resp => resp.json()),
        fetch(`/restconf/schema/${module}/`, {
            method: 'GET',
            headers: {'Accept': 'application/json'}
        }).then(resp => resp.json()),
    ]).then((responses) => {
        render(parent, responses[0].value, responses[1].value.module);
    });
}

// navigate thru the meta along with the object to build forms but
// as a pattern for all model driven UI
function render(parent, obj, meta) {
    const editableFields = meta.dataDef.filter(def => {
        // ignore mertics
        if (def.leaf?.config == false) {
            return false;
        }
        // would normally recurse here
        if ('list' in def || 'container' in def) { 
            return false;
        }
        return true;
    })
    // here you would normally adjust the input type based on details
    // in the 'def' object like number v.s. string, etc.
    parent.innerHTML = `
        <table>
        ${editableFields.map(def => `
            <tr>
                <td><label>${def.ident}</label></td>
                <td><input value="${obj[def.ident] || ''}"></td>
            </tr>
        </table>`).join('')}
    `;

    // recurse into lists and containers here
}

// car is not a very exciting object to edit but still demonstrates feature
load('car');

4.17 - Request filters

How to register web request filters to read custom request headers and how to relay that information to your nodes.

RESTCONF HTTP request access

This is useful for custom authorization or general need to access the HTTP request information

Steps:

1.) Register a request restconf.RequestFilter with RESTCONF restconf.Server instance

2.) Filter returns a context.Context that contains any custom data that you might extract from the HTTP request like HTTP header information, URL parameters or certificate information.

3.) Values from that context.Context will be made available to all your node.Node implementations

Example Code:

package demo

import (
	"context"
	"io"
	"net/http"
	"strings"
	"testing"

	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/parser"
	"github.com/freeconf/yang/source"
	"github.com/freeconf/yang/val"
)

type App struct {
}

var module = `module x {
	namespace "freeconf.org";
	prefix "x";

	leaf messageFromRequest {
		config false;
		type string;
	}
}
`

type testContextKey string

const contextKey = testContextKey("requestKey")

func manageApp(a *App) node.Node {
	return &nodeutil.Basic{
		OnField: func(r node.FieldRequest, hnd *node.ValueHandle) error {
			switch r.Meta.Ident() {
			case "messageFromRequest":
				msg := r.Selection.Context.Value(contextKey).(string)
				hnd.Val = val.String(msg)
			}
			return nil
		},
	}
}

func startServer() *restconf.Server {
	ypath := source.Any(restconf.InternalYPath, source.Dir("../yang"))
	m, err := parser.LoadModuleFromString(ypath, module)
	if err != nil {
		panic(err)
	}
	app := &App{}
	b := node.NewBrowser(m, manageApp(app))
	d := device.New(ypath)
	d.AddBrowser(b)
	s := restconf.NewServer(d)

	// Register a filter to peek at request.  From here you can look at:
	//  * headers
	//  * certs
	//  * url params
	grabHeader := func(ctx context.Context, w http.ResponseWriter, r *http.Request) (context.Context, error) {
		msg := r.Header.Get("X-MESSAGE")

		// stuff your data into the context and it will be available to all node
		// navigation associated with this request.
		ctx = context.WithValue(ctx, contextKey, msg)
		return ctx, nil
	}
	s.Filters = append(s.Filters, grabHeader)

	d.ApplyStartupConfig(strings.NewReader(`{
		"fc-restconf" : {
			"web" : {
				"port" : ":9999"
			}
		}
	}
	`))

	return s
}

func TestRequestAccess(t *testing.T) {
	startServer()

	t.Run("request", func(t *testing.T) {

		req, err := http.NewRequest("GET", "http://localhost:9999/restconf/data/x:", nil)
		fc.AssertEqual(t, nil, err)
		req.Header.Add("X-MESSAGE", "hi")
		c := &http.Client{}
		resp, err := c.Do(req)
		fc.AssertEqual(t, nil, err)
		fc.AssertEqual(t, 200, resp.StatusCode)
		actual, err := io.ReadAll(resp.Body)
		fc.AssertEqual(t, nil, err)
		fc.AssertEqual(t, `{"messageFromRequest":"hi"}`, string(actual))
	})
}

5 - Contributing

How to get involved.

Reporting an Issue or Requesting a Feature

If you’ve found a problem in the code or the docs, but you’re not sure how to fix it yourself, please create an issue in the Issue DB.

Have a Code Fix

Open an issue describing your fix and then submit a merge request.

Starting a Discussion

Not sure if you found a problem, have a general question or want to solicit feedback on an idea. Post to the discussions forum.

Documentation

(i.e. this doc)

We use Hugo to format and generate our website but there’s a good chance that if your contribution is fairly typical, then you can just markdown and not have to install Hugo locally.

Other

Email Douglas Hubler

5.1 - Roadmap

Always more work to do.

FreeCONF is a production ready library, but there a lot more things to add. FreeCONF is driven by you and your needs so speak up!

Support for more languages - Only Go is supported today but Python is in the works. FreeCONF would like to add support for more languages Java, Rust, Node, C# and Ruby.
Support for more parts of the standard - YANG Push would be useful.
Support for additional communication protocols - RESTCONF and gNMI are great first protocols but adding more would increase tool support

6 - Contribution Guidelines

The usual guidelines, but guidelines none-the-less.

We love your input! We want to make contributing to this project as easy and transparent as possible, whether it’s:

Reporting a bug
Discussing the current state of the code
Submitting a fix
Proposing new features
Becoming a maintainer

We Develop with Github

We use github to host code, to track issues and feature requests, as well as accept pull requests.

We Use Github Flow, So All Code Changes Happen Through Pull Requests

Pull requests are the best way to propose changes to the codebase (we use Github Flow). We actively welcome your pull requests:

Fork the repo and create your branch from master.
If you’ve added code that should be tested, add tests.
If you’ve changed APIs, update the documentation.
Ensure the test suite passes.
Make sure your code lints.
Issue that pull request!

Any contributions you make will be under the Apache Software License

In short, when you submit code changes, your submissions are understood to be under the same Apache 2.0 License that covers the project. Feel free to contact the maintainers if that’s a concern.

Report bugs using Github’s issues

We use GitHub issues to track public bugs. Report a bug by opening a new issue!

Write bug reports with detail, background, and sample code

Short is fine but Great Bug Reports tend to have:

A quick summary and/or background
Steps to reproduce
- Be specific!
- Give sample code if you can.
What you expected would happen
What actually happens
Notes (possibly including why you think this might be happening, or stuff you tried that didn’t work)

Use a Consistent Coding Style

All Go code should be run thru gofmt to comply with Go code formatting standards.

License

By contributing, you agree that your contributions will be licensed under its Apache License.

References

This document was adapted from the open-source contribution guidelines for Facebook’s Draft and Briandk

Documentation

1 - Overview

1.1 - What is it?

1.2 - How does it work?

1.3 - Why a standard?

Aren’t defacto standards good enough?

Additional reading

1.4 - Why the project?

Current systems

Terms

Obstacles for DevOps

Obstacles for Application Developers

Obstacles for Software Vendors

The ideal system!

Reliefs for DevOps

Reliefs for Application Developers

Reliefs for Software Vendors

The catch

Agreeing on a standard

Capable enough standards

Implementing the standard

1.5 - What can it do?

Ideal

Just a few example use cases

1.6 - Project history

2 - Getting Started

Step 1.) Get the source

Step 2.) Create a basic YANG file

Step 3.) Create an application

Step 4.) Create a startup config

Step 5.) Startup your application

Step 6.) Test your API

Done

3 - Reference

3.1 - YANG primer

Data Definitions

module

leaf

all possible leaf types

container

list

container/leaf

leaf-list

rpc

rpc/input/output

container/action/input/output

notification

notification/leaf

anydata

Organizational

group/uses

choice/case

typedef

Metrics

Constraints

number types

string types

Extensions

More

Example YANG files

3.2 - Basic components

3.3 - Using RESTCONF

Adjustable data granularity

Methods

URL

Custom URL params

Examples

Events

Subscribing to events in web browser:

Subscribing to events in CLI

Examples subscription paths:

More

3.4 - Node Development

3.4.1 - Interface

3.4.2 - Edit lifecycle

Edit: Delete a node

Edit:Create a new node thru a target node

Edit: Edit leaf on a node

3.5 - Resources

FreeCONF