Allow applications to use resources from other applications

[andyjeffries]

@alejandrojnm is trying to implement Hasura as a marketplace application, but it requires a PostgreSQL database.

First thoughts

My original thoughts were to allow passing of configuration items from one application to another. e.g.

configuration:
  DB_USERNAME:
    label: "Access key"
    value: "CIVO:APPLICATION:POSTGRES:ROOT_USER_NAME"

However, I don't think it's nice if Application B can pull the authentication details from Application A.

New plan

So now I'm thinking about a slightly nicer interface where applications can provide resources. So PostgreSQL would define something like this in the manifest.yaml:

resources:
  provide:
    - database

and it would have a new resource.sh script which takes two command line positional parameters, the action of either create or delete (for later uninstalling) and a name, which will by default by the application followed by the resource name lowercased. It will output a single line containing the details required to access the resource (usually a URL):

$ resource.sh create hasura-postgres_db
postgres://RANDOM_USERNAME:[email protected]:5432/hasura-postgres_db

Applications that require a database would request one by doing:

dependencies:
  - postgres
resources:
  require:
    POSTGRES_DB:
      application: postgres
      resource: database

The output of the resource.sh script would then be injected in to $POSTGRES_DB the same as if it had been defined as a configuration setting normally.

Would welcome any feedback from anyone writing or thinking about writing marketplace applications.

[andyjeffries]

However, I'm not 100% sold on resource.sh and resources as these may be confusing with Kubernetes resources. This is really just an issue to track the considerations and plan.

[Lazarus404]

I think you're just swapping apples with apples. Couldn't you require a secret reference in the install? Then it's up to the user to supply name and key.

[andyjeffries]

Hi @Lazarus404, thanks for your thoughts.

Another user @hazsetata has asked for functionality to be able to prompt users for values during the install for another application, so we need/want to find a way to do that.

However, if we only did it that way - in this case if you wanted to "one click install" Hasura, it wouldn't be possible. You'd need to install Postgres, then fire up the psql client to create a database and user, then use those details when you go back to install Hasura. You couldn't do it all in one step.

The mechanism I describe above allows you to one-click install Hasura, and have it automatically create a database/auth for itself in a single step. And I would feel it's not swapping apples with apples because in the early thoughts, the Hasura application would need to get the root auth details to do it (giving it potentially access to all DBs), but in the latter proposal, the Postgres is responsible with its own script for creating resources and never gives out the root details.

However, there's no reason you couldn't do it the way you suggest currently by requiring users to manually create the database/user and insert the values in to a K8s ConfigMap which the Hasura application would then use. But I'm trying to (where possible) maintain the "I want to have a play with X, on Civo I can launch a single node (or bigger) cluster and in a couple of clicks have it up and running to play with" ethos.

[rhyspowell]

I can see from some of the market place apps there is a mixture for helm and k8s yaml. I also have checked, there is no Hasura chart but wouldn't it make sense to follow helm as the standard? That way you have defaults for one click deployments but the ability to override things for those who have more experience/specific needs? THis could then apply to anything else going forward?

[andyjeffries]

Sorry, I think you may have missed something @rhyspowell. We have a use @alejandrojnm that wants to make this chart, but it needs a PostgreSQL database. Whether he uses Helm or k8s yaml is irrelevant to the point of how do you get access to a PostgreSQL database without needing the entire root details for PostgreSQL.

[rhyspowell]

With the chart, you can have it install the postgres database for your one click install, I can see there is a method in place already that will shows the root password on the current postgres one click, you would just add the Hasura info as well.

Once the way of prompting for details is figured out, thats where using helm comes in as you can set the flag to skip the postgres install just use the supplied details to install where the user wants with the creds they want.

[andyjeffries]

Let me explain another way. I want users to be able to join #KUBE100 or just use the service when it comes out and think "Hmmm, Hasura looks cool - I'd like to have a play with that":

civo k3s create --wait --save --applications=hasura

And be ready to play with it!

In order to do that it needs to a)create a new cluster, b)install the dependencies of Hasura (PostgreSQL) and then c)install Hasura itself. So we need some automated way of Hasura having a PostgreSQL database - and it feels like Hasura (or at least Hasura's Helm chart) having access to PostgreSQL's root authentication is not the best way to achieve this.

Sure, there are manual steps to work around this, but I'm trying to always maintain that "one command line to install anything" ethos for the Marketplace. There may be times we have to prompt a user for details (e.g. GitOps asking for the Git URL as @hazsetata wishes), but I don't think it should be a start, user has to take some intermediate steps (create database, create user, pass details back in), finish type of process.

[rhyspowell]

so, I'm not sure what process the above command kicks off but assuming its a script, so something like

if no args
  hasura_pw=`date +%s | sha256sum | base64 | head -c 32 ; echo`
  helm install --name harsurapostgres --set postgresqlUsername=harsura,postgresqlPassword=$hasura_pw,postgresqlDatabase=hasura,<other settings as required>
  db_endpoint=<get that data>
  username="harsua"
  dbname="harusa"
else
  echo " whats the db address:port"
  read dbendpoint
  echo "whats the user name"
  read username
  echo "whats the password"
  read harusa_pw
  echo "whats the db"
  read dbname

helm install --name hasura --set username=$username,password=$password,database=$datbase,dbEndpoint=$db_endpoint stable/harusa

there is even the k8s yaml here https://github.com/hasura/graphql-engine/tree/master/install-manifests/kubernetes

you can then show the values that you think people might need as required

[Lazarus404]

So, I think we can make some presumptions. If Postgres is installed, it would make sense that it uses a Secret. The install script can detect a PG install. If it doesn't find one, it prompts install Postgres Pod? and does everything automatically. If one is found, then it can ask provide Secret reference option, instead. At that point, if there is no Secret (the user is just trialling things), then the user has time to add one before continuing.

[andyjeffries]

The echo/read/prompt bit is what doesn't work - the scripts execute on the k3s cluster itself, so you don't get to prompt the user for anything as it installs. We can work around that by having a list of "ask for X, Y and Z" and insert them into the script, but that's still a bit yucky (IMHO) compared to the solution of having an application that can create "things" (e.g. databases) be responsible for having some automated way of creating them and returning back a unique "thing".

It would make my one line user experience from above into:

civo k3s create --wait --save --applications=hasura --application-config-hasura-db_username=hasurauser --application-config-hasura-db_password=foobar --application-config-hasura-db_database=hasuradb

And something would still have to be responsible for creating the database.

[rhyspowell]

thats only needed when the user has already take the action to do the postgres bit, else its all invisible to the user. If someone is that far down the path, I'm pretty sure they would be happy applying the args as required

[rhyspowell]

the only thing I am unsure of in the above is how the db endpoint gets exposed so you can pass that down as a variable

Also updated above to show variables being passed down from either option

[andyjeffries]

@Lazarus404 we could do that, but the only secret in the Postgres Application (Application is a Marketplace Application, compared on the underlying application itself) is the root authentication credentials, which brings it back to my First Thoughts in my description - although you're at least now pulling them out of Kubernetes. With the downside that if the field is encrypted there (bearing in mind we want the mechanism to work widely, not just for Postgres) you'd have to find a way to unencrypt it and you'd have to know the internals of the secrets used by those application rather than just using nice $SOMETHING values in the YAML or script files and have them inserted. https://github.com/civo/kubernetes-marketplace#customising

[andyjeffries]

I think I'm still missing something @rhyspowell.

So if you do:

  helm install --name harsurapostgres --set postgresqlUsername=harsura,postgresqlPassword=$hasura_pw,postgresqlDatabase=hasura,<other settings as required>

How would it create the database and user/password in PostgreSQL, without having some authentication details to connect to it in order to be able to create them?

Or something will still have to create them (in the middle of a one command install), which brings me back to what I am trying to solve :-)

[rhyspowell]

That command does it for you, create the user hasura, the password as set from the variable and the database harusa, the chart does it all for you.

Sorry just noticed that that line is missing the stable/postgres from the end of the command

[Lazarus404]

What if the install always installed the dependency Pods, but in a different namespace? That way, if the user already has the dependency, they can just reconfigure the Pod definition and remove the unused dependency... It keeps it simple, at least.

[andyjeffries]

That command does it for you, create the user hasura, the password as set from the variable and the database harusa, the chart does it all for you.

I don't understand how it can do it for you. How can it create a user, password and database WITHOUT being able to login to PostgreSQL itself? Or without pulling the root secret out of a ConfigMap or Secret, which may change in the future.

[rhyspowell]

So a deeper dig https://github.com/helm/charts/tree/master/stable/postgresql#configuration they are the admin creds, thought they were additional creds sorry, having said that, of a one click deployment, would that make a difference? If yes, then it will get very yucky with currently the only option I could think of is a short run of a container with psql installed and create the user, password, db and access through that. Under the assumption that psql isn't installed on any of the cluster nodes.

Sadly I don't have the time to the next to days to know up a PoC as I'm still not familiar with how the scripts get kicked off in the first place

[andyjeffries]

I think maybe the best bet might for me to implement what I'm suggesting, and then I can share the code of how it works/hangs together for Marketplace Application developers.

[ssmiller25]

So tripped across this while trying to figure out how to package a marketplace app that would require a database backend. So what if the back-end resources like MySQL/Postgres/etc were provided as Kubernetes Operators? My idea would be to provide something like the KubeDB Operator that would only deploy the operator framework for creating databases. Then a dependent application would provision the database they want leveraging that operator. That way there are no credential passing issues, and the application itself would have more control over how the database would be provisioned. I might sketch out a KubeDB and dependent application (maybe Wordpress, since it's well understood and would seem like a great add to the marketplace)

[andyjeffries]

I like the idea for using a Kubernetes Operator, however, how would the KubeDB operator know the credentials? Would you bundle say the MySQL operator with MariaDB - or have KubeDB be a dependency of MariaDB and somehow have it know to accept resources for MySQL but not PostgreSQL. Definitely interested in the concept though. Seems a more native way of doing this.

[ssmiller25]

My thoughts is that the originating app would use a CIVO:ALPHANUMERIC(num):BASE64 to create a random password in a secret, then pass that both to whatever operator is creating the database, as well as the application itself for configuration.

My thoughts is any of the databases that bundle operators would only deploy the controller for the operator, then the originating app would request the database it needs when it comes up (and would be deployed at the time the application is deployed). For instance, if my application needs MariaDB, then I would include a CRD for MySQL as part of my manifest, and the KubeDB operator would build out a MySQL database at that time.

I'll try to sketch some code up for the marketplace if I have a chance. I like the idea that if there are inter-app dependencies that we use operators to create/manage those dependent resources. Although that also would make a presumption for application authors that, if they are building an app that could be a dependency, that an operator is also provided. Perhaps that's a bit of a large ask.