Storage of contextual configuration data #1349
Comments
|
I definitely see the value in adding something like this to NetBox. The tricky part is going to be implementing it. We want to provide something more valuable than a vanilla key/value store: as mentioned, we should be able to tie values to NetBox objects like regions, sites, etc. But we also want to ensure we maintain sufficient flexibility, not knowing what people might want to use it for. We also don't want to duplicate the functionality provided by custom fields. At a higher level, this is going to tie in somewhat with configuration management (e.g. the ability to render device configurations using templates and NetBox data), but I haven't given that topic much thought yet. |
|
I agree that great care needs to be taken to ensure we don't just implement another version of I'm not an expert by any means, but would just like to contribute back for all the help that has been provided. |
|
@bdlamprecht What prompted you to open this issue? Are you interested in generating device configs using this data as context, or did you have something different in mind? |
|
Yes, I'm working on a POC using NetBox as a "Source of Truth" for Ansible which will then create / manage configuration templates for my large enterprise customer (1000+ templates) and push those changes onto the device using NAPALM (or possibly another Ansible "module"). Our "service delivery" team that installs the so-called "templates" which are created by hand rarely get them correct (which is understandable as they are, in my opinion, unnecessarily complex). This project I'm working on is to try and automate and orchestrate that process. |
|
Here's an idea: We create a For example, I create a ConfigContext assigned to a region called "North America" with the following: Then I create a second ConfigContext for a specific site in that region with special DNS servers: The context available for a device in that site would render as: We could make this available via the API, but the next stop logical step would be to render device configs from templates directly within NetBox. |
|
Great, I love the idea of the "more-specific" information overrides the other "generic" information. I'm confused by your last statement however... The POC I'm working on now would store the templates in GIT so you can generate the configs and push them to the devices based off of which "release" you have "checked-out" in GIT. I don't feel the need to duplicate efforts by other tools already in existence. All this being said, it is great work and, once again, I really appreciate all of the work that you're doing with developing NetBox. |
Not store them natively, but perhaps read them from a repository.
What are you using to render the templates? Presumably you're pulling data from NetBox via the API? |
|
Well, don't judge me as I'm new to the developer world and I'm fairly sure there's a better / more efficient way to accomplish the end goal I have. Anyways, I'm using the Ansible |
Hey, me too! 😄
That seems like a very reasonable approach. So it seems like in your case just making additional context data available via the API would be sufficient. |
|
Yeah, I think that would work. I'm not sure if I made this clear, but it would be important, at least for my scenario, to have the new As an example, something like BTW, it doesn't seem like you're new to the developer world based off of how quickly you implement new features and update NetBox, but if so 🥇 |
|
So this is really exciting to see. I have been building this sort of thing in other places in my automation suite integrating with netbox. I have been really flustered because IMHO the best place for this is inside of netbox itself. That being said, I do have a few ideas (and would love to help see them through)... Today I store this data either in group vars in ansible playbooks (for generic role based config) and a hacked together CMDB (for device specific config).
@jeremystretch I think it is best to store this in netbox. I would be fine storing this as actual yaml (as flat files) or as json in postgres. That is flexible though; if the solution is elegant, I don't really care. I would want to see the ability to assign a
Device type and role are the interesting ones as they would allow me to derive use case for a device (as has been previously discussed in the Network to Code slack). The hard part with allowing all of those is defining the hierarchy of precedence. To that end, I think the solution is allow it to be configurable rather than hard set. So perhaps have a default hierarchy but allow the user to modify it if the default presents conflicts for their use case (I assume it would, but maybe I am over thinking this part). |
I was referring to configuration templates. Those won't be managed by NetBox directly. |
|
I'm trying to decide how best to implement this in the API. Option A: Always include config context with the DeviceSerializer (including when listing multiple devices). This seems terribly inefficient. Option B: Leave DeviceSerializer alone and create a detail route to retrieve the config context for a particular device, e.g. Option C: Implement a request parameter like Open to other suggestions. |
|
I would not do option A. I only want the config context in specific situations, not all. I would like to see a normal set of endpoints for This plays into my earlier comments on dynamic hierarchy. In this model the dev/api user gets full control in how they interpret the hierarchy. Obviously this means if they want to build a full config, they may need a series of api calls all at once (is that really a bad thing?) to get all the contexts for a config. To that end, I also like options B which would implement the "default hierarchy" I proposed. Just my thoughts. |
|
I agree that Option A is definitely a no-go for me. Not necessarily in my scenario I detailed above, but I can foresee some circumstances where not all Now that I think about it, perhaps a mixture between both Option B and Option C would be ideal if the API could be structured in a way where you could do something along the lines of As @jeremystretch mentioned, that WOULD set a "precedence for adding/removing fields dynamically". However, I believe that it would add value and it would be up to the end-user to implement it correctly and understand the pros and cons to structure their data correctly. Some documentation on the "whys" would probably be required 😉 and I'd be willing to help write it once I understand the structure that is implemented. Again, these thoughts are coming from a newbie to the developer world so this might be fairly complex to do or not even possible at all, so take my comments with a "grain-of-salt". |
I would vote against this, as it makes the api a lot harder to query. It seems rather simple to me to only use the values you need on the client side. A system such as the proposed would only make sense to me when one would add a lot (like thousands) of such context fields, because then there would be significant overhead in fetching the values from the database and also in transferring that data to the client. Option C is my favourite anyway. It is simple enough to implement in an API client. It could even be added to the query for many devices, i.e. |
|
While I'm not opposed to Option C, I don't understand the following statement:
In the example I used above you could always still query the API via However, if that is too complex to implement on the server-side, I'd be okay having Option C as it follows the core guidelines of NetBox:
Again, another disclaimer, this is coming from a recent transfer from the networking world to the developer world (DevOps) so I apologize in advance for any ignorance that I'm showing 😃 . |
|
I think I prefer option B. It might help to better explain the way I see it working. Let's say you've defined three ConfigContext objects: Global Site A Edge Router You would retrieve these ConfigContexts via an API endpoint at Now, rendering the config context for a given device is different. If we want to get the rendered context for a device (e.g. an edge router in site A), we'd request Notice that the DNS values from the global ConfigContext have been replaced by the ConfigContext for site A. It's important to point out that this request does not return a real ConfigContext object, but rather a dictionary compiled from all ConfigContexts applicable to this device. |
|
Yeah, I see where you are coming from. This statement is going to show my inexperience, but in all reality, the chance of having too many The way you explained the rendering above makes sense to me as well. As you mentioned previously, this would allow the current API to continue to function as it has while also allowing an addtional API call to So although my vote doesn't mean much since I'm not the one developing it, I'm onboard for Option B. |
|
@jeremystretch I agree with this implementation. It is important to me to always maintain |
|
I don't follow the argument for option B completely when it comes to We also define e.g. sites in I understand, that including the config-context by default is not desirable, because it causes overhead to calculate it's values, but that's why Option C suggests the '?include_context=1' flag. Myself, I'd rather do one call to the API, than two. At times it's hard to correlate two calls. Also, the second call could fail due to some error condition (e.g. network) that I will have to take care of. Maybe I could share here one use-case we're investigating and where this feature would become very useful to us: Creating our 'static assignments' dhcp configuration file. For our solution, we'd also leverage the hierarchical values, e.g.: Global: Region 1: Region 2: Device 2 (overwrite default value): |
I think I now get what you would like to express here: But I still think, that from an API-user's perspective, it'd be favourable to get all required information about an entity in one call to the api, rather than two. (Especially, if I would have to request information on more than one entity, as I have made a case for above.) |
Yep, exactly.
I agree that it would be ideal, but consider that multiple requests are needed anyway to retrieve interfaces, IP addresses, VLANs, etc. I'm not sure whether it would be practical (or efficient) to try and return all of that in a single response. |
That would only be true for that exact case I made above. One might only need the information belonging to a device, e.g. to populate a cloud-init user-data template. Another argument I'd like to make in favour of Whereas |
The queryset gets pretty unwieldy when you start pulling devices in bulk. We'd need to prefetch related ConfigContexts for all devices in the list. Since a ConfigContext can be related by region, site, device_role, tenant, and/or tenant_group (plus global contexts), this becomes very inefficient. Additionally, needing to render contexts for multiple devices in a single request seems like an uncommon use case. |
Sure, I agree. That's why it's never enabled by default.
It's all the same inefficient when I would query each device one-by-one. Nevertheless, I was just saying that I would prefer the |
|
Does anyone see it as a nice-to-have to be able to concatenate the content of different inheritence levels instead of getting the most specific value (to whatever preference agreed upon)? For example: Site1: When querying the device could render: Since there doesn't seem to be a standard way of doing this with YAML, a syntax would have to be come up with including the 'fixup' code to flatten the aggregated content? Example taken from |
|
@sdktr The Stack Overflow post you linked mentions the YAML merge key. Seems reasonable, but I'd have to see how involved supporting its logic would be. |
|
Interesting feature. I can see this being very useful. With regard to allowing people to store the data how they want in #1349 (comment) if you used the PosgreSQL JSON field type, then you get free form JSON, and could convert to YAML (or whatever) based on what is requested from the API. This would also make it much easier to display, edit & validate this content in the UI in a nice manner, rather than just a giant text field. PostgreSQL also has nice features now for querying for data inside the JSONB data type fields, so you'd be able to support queries for things in this data in the future (or for people using SQL as their API). More info on how PG handles JSON is here: https://www.postgresql.org/docs/9.6/static/datatype-json.html |
|
Any updates on this request? I seem to recall that there was a alpha version working in one of your branches. |
|
I think this would be an extremely valuable feature that would further Netbox as the SoT. As more organizations automate their environments, data becomes essential and defining a data model for managing device configurations is required. In our case, we are wanting to manage thousands of devices with Ansible and Ansible Tower/AWX. We have chosen to align with OpenConfig to build our own data model. We would like to store and associate the data in a hierarchy so we can align it with Ansible's group and host variable structure and minimize data duplication (DRY). Option B that Jeremy mentions would work well for this use case. It would be very interesting to see if YANG or JSON data models (like OpenConfig) could be broken apart and represented within Netbox directly to accomplish the task of loading your organizations data. It would be nice to see a default hierarchy of precedence that could be customized based on what the customer requires: Tenant Making this data accessible via the API would we critical. With it, customers can create their own dynamic inventories and complete device data models to use with their config management tool of choice, e.g. Ansible, Salt, etc. Configuration templates should not live in Netbox as these would be managed by the customer in git repositories. Another feature possible with Netbox however would be a front end for managing these templates and integrating with git to do so. |
|
@jeremystretch I know you are very busy, but is there anything I can do to help expedite this request? As I stated previously, I don't want to be "annoying annoying or in any way demanding", however, the "proof-of-concept" that I've been working on for a few months is highly dependent on this functionality. What do you think would be of the most help to you?
Really interested in getting this working within the next few months. |
jeremystretch
added
the
status: accepted
|
I've been thinking about this more lately. I'm hung up on the hierarchical approach because it doesn't allow the assignment of values along two dimensions. Simple HierarchyFor example, say we implemented a hierarchy like this, wherein data defined at each layer supercedes any data with the same key defined at the layer above it:
Contextual data could be defined directly on each object as a JSONB field. But what happens if you need to assign, say, We can define a unique value for each role and site, but one must take precedence over the other. In this example, role would win because it is below site in the hierarchy. We can work around this by defining the value for the most common of the three roles at the site level, and then defining values for each of the other two roles directly on the individual devices, but this is redundant and error-prone. And it becomes even less efficient if another dimension like platform or device type is introduced. Alternative ApproachInstead of defining context data directly on each object as a field, we can introduce a ConfigContext model which has ManyToManyFields typing it to sites, roles, etc., as well as a JSONB field to store the data. In our example above with four sites and three roles, we would create 12 ConfigContexts:
To retrieve all ConfigContexts matching a particular device, we would query like this: This will return all ConfigContexts assigned to the device's site and/or role (as well as any which are not assigned to any particular site or role). (Of course, we can match on other attributes as well, not just site and role.) This approach should be fine for individual devices, but doesn't scale well when retrieving multiple devices. This is especially true if we decide to filter across multiple tables, like One interesting but somewhat scary approach would be to render the config context for every device each time a ConfigContext is modified, and store the rendered context locally on each device. I'm not sure yet how involved this process would be, but it's worth considering that context is likely to change far less often that it will be read. And pre-computing the rendered context for each device would do wonders for performance, particularly in retrieving many devices via the API. I'm curious to hear any thoughts others have before I go any further with this. |
|
Following up on my comment above after some discussion on NetworkToCode. I envision the ConfigContext model looking something like this: We may decide to add other fields like Each ConfigContext is assigned its relevant sites, roles, etc. The Additionally, we'd add a Adding, modifying, or deleting a ConfigContext triggers rendering of configuration data for every device. (If we adopt a message queue as has been suggested for #81, this operation can be handled in the background.) This would involve: 1. Retrieve all unique sets of devices Every device which has the same site, role, tenant, and platform (in this example) will receive the same rendered configuration data. Thus, we only need to render the context for each unique set of devices, rather than for each device independently, using the following query: 2. Render the context data for each set For each set of devices, we query all relevant ConfigContexts, ordering them by weight and time of creation. A dictionary is created and updated with the contents of each ConfigContext in sequence. 3. Update all devices in the set at once Using Obviously, lots of testing is needed, but it seems plausible. We do potentially end up with quite a bit of redundant data, but this denormalization buys us the ability to return all config context for a device directly from the device table. This means we can pull down the config context for many devices at once with a negligible performance penalty. |
jeremystretch
changed the title
|
Just a thought: should this 'configcontext' and 'custom fields' functionality be merged? To mimic the 'custom fields' functionality as it is today it would just be a one-to-one mapping to the relevant object that stores it (like the simple configcontext proposed before jeremy pitched the configcontext as a seperate model) An additional benefit of upgrading the 'custom field per table' to the 'context is a seperate model' is that migrations of the database are simplified. There would be just a seperate table/model with all the custom stuff in there instead of each installation having their own tables with custom columns. Quoting @jeremystretch model for this:
|
Each serves a different purpose and complement one another. A custom field allows a user to specify (even require) a particular value per object. They can be assigned to most primary objects in NetBox. A custom field works well if you need to track values which are generally unique to each object. A config context is intended to provide additional, optional, free-form data to a set of devices. This would better accommodate data that is the same for each device that meets an arbitrary classification (e.g. all access switches at site D). Also note that contextual data by its nature cannot be declared as mandatory. |
|
I understand that both might have their own primary usage, especially the way they are positioned today. The reason I came up with this is because of how you proposed to make 'configcontext' it's own seperate model. If I understood your intentions correctly, you did this to have a more flexible combination of data end up at the device level (instead of just inheritance in whatever order). The way you set up your model is that it could have many-to-many relations to one or more objects. That same datamodel seems to provide a good (imho: better) way of storing the simpler 'custom fields' that have a one-to-one relation to an object today. The 'data contract' for these 'custom fields 2.0' can be stored in a seperate table. This way mandatory data can be checked for at the object level (or whatever the data construct dictates). Besides 'configuration data' I have other data that won't fit in the regular fields of Netbox objects today. My initial thought would be to store json in 'custom fields' (if possible?) and use basic inheritence to end up with a complete json object at the requested object. |
|
Work on this feature has been merged into |
Issue type:
Feature Request
Python version:
2.7.12
NetBox version:
2.0.8
With the goal of NetBox being THE "Source of Truth" for the desired state of a network, I noticed that it does not currently have a location to store values for IPv4/IPv6 addresses of DNS, NTP, TACACS, SYSLOG, SNMP, and other servers which support network devices (as well as servers).
While thinking through real-world scenarios, these values are usually per region and/or per site, so
custom_fieldscould be used, but I think it would be applicable to a large portion of NetBox users to dedicate the time to implement this update to the ORM schema.For my thinking, it I believe it would be best to possibly create a new section in
IPAMto store thesekey:valuepairs which could then be tied to eitherregionsorsitesto minimize querying the DB so often, but it is highly likely that some other place would be more beneficial.Pinging @jeremystretch for his thoughts.
The text was updated successfully, but these errors were encountered: