File ingestion - JQ manipulation | Voters

File ingestion - JQ manipulation

complete

Zbigniew M (ferror)

Currently, you can ingest files using the format
file_property: file://composer.lock
Being able to apply JQ at the file ingestion stage, will allow you to ingest less data to Port for a cleaner and faster portal, for example:
file_property: file://composer.lock | .myProperty

Created by Omri Gez

December 27, 2023

Dudi - Port team

marked this post as

complete

We just released the ability to Map GitHub file content to the catalog!

You can now bring GitOps to life with

json

and

yaml

mapping, enriching your catalog with any data you already have in your repositories:

Use JQ to map specific fields to your catalog entities. For example, map the “package manager” from a
package.json
file to a service entity
Create and update multiple entities based on arrays in files. For example, map all dependencies from a
package.json
as an entity per dependency

See docs to learn how to map file content from GitHub.

The feature is currently available for Port's GitHub integration. In the future, we will add this capability to all Git providers.

Sebastien Foo

This is great news ! how do we activate it cause I don't see the "file" kind in my github exporter ?

Gur Shafriri

marked this post as

planned

Time has passed, but we are now designing the solution for this.

The current plan is to add a "file" kind, supporting querying json or yaml files with JQ/YK, for one of the two:

setting entity property based on the file content
creating new entities based on the file content

Sebastien Foo

We could also set meta-properties such as $team from the content of any yaml file

Sebastien Foo

great request,
It will also help ingesting specific customer (like we have at Qlik) data model. We already have our own data model around yaml files.
We have one file at the root of the repo that contains the component (similar to a service) name, slacks channels, teams, champions, and also some other metadata used by our CI/CD. I was able to load this file using one property and then use "Calculation" fields types.
But we have also a separate yaml file to define Component dependencies.
here is an example of the componentDendencies.yaml file
components:
- component-a 
- component-b
and for this I cannot use "calculation" fields. 
So this requests would be a fantastic improvements, not only for "properties" but also for "dependencies" settings.