Example gitlab file for docker in docker file with authentication

I ran into a problem in a special docker environment. The docker within this env doesn't allow downloading images from central registries. To bypass this I wrote a dind gitlab-ci.yml which starts a docker in docker image to download the required image and pushes the image into the custom registry. The dind needs to authenticate to push an image to the custom registry or to consume the image from the custom registry. 

Here ist a generic sample gitlab-ci.yml, which downloads and pushes an image. To run this you need the following ci vars configured within gitlab

IMAGE_NAME        : Name of the image to fetch i.e. trion/karma

IMAGE_VERSION   : Version of the image i.e. latest

DOCKER_AUTH_CONFIG: String with a valid docker auth config for your custom repo

CI_REGISTRY: Link to your docker registry

image: docker:20.10.24

variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""

services:
    - name: docker:20.10.24-dind
        entrypoint: ["dockerd-entrypoint.sh"]
        command: ["--insecure-registry", "custom.registry.mycomp.org:443"]

before_script:
    - docker info

# job for fetching any image in the var $IMAGE_NAME:$IMAGE_VERSION

deploy-generic-image:
    stage: build
    when: manual
    tags:
        - docker-in-docker
        - PROD
    before_script:
        - mkdir -p ~/.docker
        - echo $DOCKER_AUTH_CONFIG > ~/.docker/config.json
    script:
        - echo "Pulling $IMAGE_NAME:$IMAGE_VERSION"
        - docker pull $IMAGE_NAME:$IMAGE_VERSION
        - docker images
        - docker image tag $IMAGE_NAME:$IMAGE_VERSION $CI_REGISTRY/$IMAGE_NAME:$IMAGE_VERSION
        - docker images
        - docker push $CI_REGISTRY/$IMAGE_NAME:$IMAGE_VERSION



If you don't like the docker_auth_config you could also add the login info in the before_script section like that

before_script:
    - echo "$CI_JOB_TOKEN" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

But to consume the image within your gitlab-ci.yml you will need the docker_auth_config. 

Sample Jenkinsfile

This is a sample for a Jenkinsfile with following features in use:

• REST list parameter plugin usage for retrieving milestones from a gitlab project
• Parsing JSON from text parameter with Pipepline Utility Steps plugin to get the iid of a milestone
• Calling shell script with credentials
• Save passing credentials to shell script

pipeline {
  agent any
  parameters {
string(name: 'app', defaultValue: 'sample', description: 'Name of the app')

RESTList(
      name: 'MILESTONE',
      description: '',
      restEndpoint: 'https://git.gitlab.com/api/v4/projects/234/milestones',
      credentialId: 'CREDENTIAL_ID_IN_JENKINS',
      mimeType: 'APPLICATION_JSON',
      valueExpression: '$[*]',
      displayExpression: '$.name',
      cacheTime: 10,    // optional
      defaultValue: '', // optional
      filter: '.*',     // optional
      valueOrder: 'ASC' // optional
    )

    choice(name: 'sampleChoice', choices: 'True\nFalse', description: 'Well just a sample choice. First entry is defaultValue.')
    
  }
  stages {
stage('Prepare landing zone') {
steps{
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: 'CREDENTIAL_ID_IN_JENKINS', url: 'git@git.gitlab.com:sample/repository.git']]])
}  
}
    stage('Excute whatever you like') {
        
      steps {
script {
// parse the milestones retreived from gitlab
ms = readJSON returnPojo: true, text: env.MILESTONE
// save the milestone iid within the env
env.milestoneId =  ms.iid
}
withCredentials([string(credentialsId: 'CREDENTIAL_ID_IN_JENKINS', variable: 'accesToken')]) {
println accesToken: accesToken
println app: app
println sampleChoice: sampleChoice
println milestoneId: milestoneId
sh ('printenv')
dir('out') {
    sh ('pwd -P')
}
    sh ('echo $accesToken $gitLabProjectMode $app $sampleChoice $milestoneId')
}
      }
    }
  }
} 

Update to Spring > 2.5

 I've recently updated one of our spring boot apps. We've used Spring Boot 2.4 and I update to the latest available 2.6. After restarting the tests and test application I noticed a problem with the initialization of the h2 db. 

We tend to use the former spring datasource properties 

spring.datasource.schema=classpath:dev/schema.sql

spring.datasource.data=classpath:dev/data.sql

which are renamed into spring.sql.init.*. So I also renamed them, but nothing happened so far. Neither schema nor data.sql gets called during startup. After reading the changes I noticed a hint in the Init Docu that you should not use the init scripts with higher level db migration tools like liquibase and flyway.

We are using flyway and that was causing the issue. Form Spring Boot 2.5.1 on the init script will be ignored if you are using flyway or liquibase (spring boot doc).

So I changed the way we init the dev and test h2 db. I set the 

spring.sql.init.mode=never

and removed the schema.sql and data.sql. To initially create and fill our db with flyway I added to flyway snippets using the flyway hook for beforeMigrate. As we have to script I used the standard flyway notation for the two scripts like the following

beforeMigrate__01_schema.sql

beforeMigrate__02_data.sql

After that change fly way runs the two scripts before starting the migration when using the h2 db. I only added the two script to our flyway h2 snippets.

 

Self signed certificates and Oracle 19

Problem 

Starting with Oracle 19 you need to do some additional steps to allow Java calls within your Oracle 19 db to access self signed web services or other ssl/https resources. You'll notice the problem with error like the following while calling an https site from java within Oracle 19:

ORA-29532: Java-Aufruf durch nicht abgefangene Java-Ausnahme beendet: java.rmi.RemoteException: java.rmi.RemoteException:; nested exception is: 

                HTTP transport error: javax.xml.soap.SOAPException: java.security.PrivilegedActionException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Solution

Copy the existing cacerts file located in /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts

to /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt

add your self signed cert with (how to get my cert from an existing web site)

keytool -importcert -trustcacerts \

    -keystore /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt \

    -storepass changeit \

    -file my_ca_cert.crt -alias myrootca \

    -v -noprompt

Load the file into oracle with sqlplus

$ sqlplus / as sysdba

SQL> alter session set container = orclpdb;

SQL> exec dbms_java.loadjava('-schema SYS -grant PUBLIC -dirprefix /home/oracle/database/javavm/jdk/jdk8 /home/oracle/database/javavm/jdk/jdk8/lib/security/cacerts.alt')

or with the CLI Tool 

$ cd /home/oracle/database/javavm/jdk/jdk8

$ loadjava -user sys@db -v -schema SYS -grant PUBLIC  \

    /lib/security/cacerts.alt

Flyway spring.jpa.hibernate.ddl-auto=validate fails with Oracle synonyms

Using Flyway alongside with Hibernate is a nice matchmaker. Within one of the Spring Boot project I noticed that the flyway validate fails. Reason was a missing view. But looking deeper into the conf shows a synonym, which is handling the view. Doesn't appear within the dev env because the devs disabled the long running validate.

They got error messages like:

Schema-validation: missing table

Schema-validation: missing column

Easy to get rid of this within a spring boot app if you know how. Here is my thought on that:

spring.jpa.hibernate.ddl-auto=validate

spring.datasource.hikari.datasource-properties.includeSynonyms=true

spring.jpa.properties.hibernate.synonyms=true

will help Spring Boot to configure the hikari pool and the hibernate source to also use synonyms for the metadata exploration.

Take care to also add config for tomcat or any other pool, if you don't use the spring default hikari pool.

Windows Subsystem for Linux (WSL)

If you need a real linux shell on your Windows 10 system follow these steps

1. Ensure "Windows-Subsystem for Linux" is active on your machine with the following cmd in a admin power shell

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

2. Restart your machine if asked

3. Download the linux distribution of your choice via wget 

Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1604 -OutFile Ubuntu.appx –UseBasicParsing

or within you browser  

https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6?activetab=pivot:overviewtab

or even from the MS App Store. 

You'll find a list of distributions here 

• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• OpenSUSE Leap 15
• OpenSUSE Leap 42
• SUSE Linux Enterprise Server 12
• SUSE Linux Enterprise Server 15
• Kali Linux
• Debian GNU/Linux
• Fedora Remix for WSL
• Pengwin
• Pengwin Enterprise
• Alpine WSL

4. Install distribution with

Add-AppxPackage .\Ubuntu.Appx

5. If you now search for Ubuntu in your start menu, you'll find your ready to rock linux console in windows

Appcelerator Titanium projects with IntelliJ

Developing Appcelerator mobile Apps is usually done within the Axway Appcelerator Studio. I'm using IntelliJ to develop most of my apps and here is the way how you could build and run you Appcelerator projects.

Open the project and create a package.json like this:

{
  "name": "my-app",
  "version": "1.0.0",
  "description": "Sample package.json for running titanium apps within IntelliJ.",
  "main": "index.js",
  "directories": {},
  "scripts": {
    "setup": "./node_modules/.bin/titanium sdk install 8.3.1.GA --default",
    "clean": "./node_modules/.bin/titanium clean",
    "build": "npm run build:android && npm run build:ios",
    "build:android": "./node_modules/.bin/titanium build -p android -b",
    "build:android:full": "npm run build:android",
    "build:ios": "./node_modules/.bin/titanium build -p ios -b",
    "build:ios:full": "npm run build:ios",
    "init:android": "npm install && npm run setup | $ANDROID_HOME/tools/bin/sdkmanager --licenses",
    "init:ios": "npm install",
    "android": "./node_modules/.bin/titanium build -p android -T emulator -C HUGO",
    "ios": "./node_modules/.bin/titanium build -p ios -T simulator -C HUGO",
    "download:android:sdk": "$ANDROID_HOME/tools/bin/sdkmanager \"platform-tools\" \"platforms;android-29\" \"build-tools;29.0.2\" \"emulator\" \"ndk-bundle\" \"system-images;android-29;google_apis_playstore;x86\"",
    "create:avd": "echo \"no\"| $ANDROID_HOME/tools/bin/avdmanager create avd -f -n Pixel_2_API_28 -k \"system-images;android-29;google_apis_playstore;x86\" -d \"pixel\"",
    "configure:avd": "for f in ~/.android/avd/*.avd/config.ini; do echo 'hw.keyboard=yes' >> \"$f\"; done",
    "prepare:env:android": "npm install && npm-run-all init:android download:android:sdk create:avd configure:avd"
  },
  "author": "Andre Dvorak",
  "license": "GPL",
  "homepage": "https://www.kambrium.net",
  "devDependencies": {
    "npm-run-all": "4.1.5"
  },
  "dependencies": {
    "alloy": "1.14.1",
    "titanium": "5.2.1"
  }
}

To start a device session with your new app just open the package.json an select

1. "setup" target
2. "android" target 

After the build you will be asked you for the emulator of choice. You could skip this by replacing HUGO with the name of your favourite emulator.