AKIvan

AWS CodePipeline Multi Account Cross Region

Tue, 09 Feb 2021 00:00:00 +0000

Why do we need this?

There are many use-cases where multi-account and cross-region CloudFormation stacks can be useful. It happens a lot when you have one pipeline but the same deployment need to be done on different region, and also to cover the concept of test and production. This is some main steps that can help you start from some Account like (Tools) and deployment goes to Test and Production Account but on two regions or at least can help you to build similar one.

AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. CodePipeline automates the build, test, and deploy phases of your release process every time there is a code change, based on the release model you define. This enables you to rapidly and reliably deliver features and updates.

So the first thing is S3 Bucket on each region that we desire to deploy. After the buckets are created in their respective region, I decided to use SSM Parameters to provide the Pipeline with the buckets. Pipeline needs 1 bucket per target region.

CodeBuild

AWS CodeBuild is a fully managed build service in the cloud. CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy.

The most relevant attributes here are the environment variables containing each region bucket and the buildSpec path which should be placed inside the source code that the Pipeline will fetch from GitHub template and have it as ready to deploy part. Prepare your environments to have S3 bucket to store the templates or source of the builds there. And if you are using encryption for it, that means you should crate KMS Keys for each region too.

In this step, you create a build specification (build spec) file. A buildspec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build. Without a build spec, CodeBuild cannot successfully convert your build input into build output or locate the build output artifact in the build environment to upload to your output bucket. The main command and stuff is happening in the build configuration section in this file.

version: 0.2

env:
  parameter-store:
    GIT_USER: "git-user"
    GIT_PASS: "git-pass"

  build:
    commands:
      - sam build -t template.yml
      - sam package --template template.yml --s3-bucket "$S3_BUCKET_region1" --s3-prefix "$PROJECT_ID/builds" --output-template template-export-region1.yml --region eu-west-1
      - sam package --template template.yml --s3-bucket "$S3_BUCKET_region2" --s3-prefix "$PROJECT_ID/builds" --output-template template-export-region2.yml --region ap-southeast-1

  post_build:
    commands:

artifacts:
  type: zip
  files:
    - template-export-region1.yml
    - template-export-region2.yml

As you can see, it’s a different template export for each region and for each region there is a separate S3 bucket to store it, this can be put as one template but if you have different templates you can play it like this.

CloudFormation

Back to the cloudformation template. From the resource perspective we will start with defining the CodeBuild service. Note here that we are using this build for out template and as mention above, we need to create environment variables for each region/S3 bucket where we going to deploy it.

BuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Description: !Sub Code Build project generated for ${ProjectName}
      ServiceRole: !Ref BuildRole
      EncryptionKey: !Ref KMSKEY
      Environment:
        Type: linuxContainer
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
        EnvironmentVariables:
          - Name: S3_BUCKET_region1
            Value: !Ref S3BUCKETregion1
          - Name: KMSKey_region1
            Value: !Ref KMSKEYregion1
          
          - Name: S3_BUCKET_region2
            Value: !Ref S3BUCKETregion2
          - Name: KMSKey_region2
            Value: !Ref KMSKEYregion2
          
          - Name: ACCOUNT_ID
            Value: !Sub ${AWS::AccountId}
          - Name: AWSREGION
            Value: !Sub ${AWS::Region}
      Source:
        Type: CODEPIPELINE
      TimeoutInMinutes: 10
      Tags:
        - Key: Name
          Value: !Ref ProjectName
      Artifacts:
        Type: CODEPIPELINE
      Cache:
        Type: LOCAL
        Modes:
          - LOCAL_CUSTOM_CACHE

CodePipeline

Then we starting with the pipeline service or CodePipeline

  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      RoleArn: !Ref PipelineRole
      Name: !Ref AWS::StackName
      Stages:
        - Name: Source
          Actions:
            - Name: Source
              ActionTypeId:
                Category: Source
                Owner: ThirdParty
                Version: 1
                Provider: GitHub
              Configuration:
                Owner: GitHubOwner
                Repo: !Sub ${ProjectName}
                PollForSourceChanges: false
                Branch: !Ref Branch
                OAuthToken: ""
              OutputArtifacts:
                - Name: SourceArtifact
              RunOrder: 1

As a first action in this stage we defined the source, GitHub. There can be different ways to defined, this is one template. Note here that we defined the output artifact that later we going to reference to. The second action is the build.

      Name: Build
      Actions:
      -
        Name: BuildAction
        ActionTypeId:
          Category: Build
          Owner: AWS
          Version: 1
          Provider: CodeBuild
        Configuration:
          ProjectName: !Ref BuildProject
        RunOrder: 1
        InputArtifacts:
          - Name: SourceArtifact
        OutputArtifacts:
          - Name: BuildOutput

The main stuff here to point are InputArtifacts and OutputArtifacts. As input, we have the source from the previous action “SourceArtifiact” and as output we have already build stuff named as “BuildOutput”. Step three, we have the deployment processes.

- Name: Deploy
    Actions:

        - Name: CreateChangeSet-region1
          Region: region1
          ActionTypeId:
            Category: Deploy
            Owner: AWS
            Version: 1
            Provider: CloudFormation
          InputArtifacts:
            - Name: BuildOutput
          Configuration:
            ChangeSetName: project-name-stack
            ActionMode: CHANGE_SET_REPLACE
            StackName: !Sub '${ProjectName}-stack'
            Capabilities: CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND
            TemplatePath: BuildOutput::template-export-region1.yml
            RoleArn: !Ref DeployRole
          RunOrder: 1
          RoleArn: !Ref PipelineRole

    # *** Duplicate / edit / delete these stacks as necessary for your desired regions ***
      # *** Change this name to match desired region
    - Name: DeployToRegion1
      # *** Change this for desired region
      Region: region1
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Provider: CloudFormation
        Version: '1'
      InputArtifacts:
        - Name: BuildOutput
      Configuration:
        ActionMode: CREATE_UPDATE
        Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND
        RoleArn: !GetAtt CloudformationRole.Arn
        StackName: !Ref ApplicationStackName
        # *** Change this to match the region of this stack, 
        # *** using the file generated in the build step
        TemplatePath: BuildOutput::template-export-region1.yml
      RunOrder: 2
      # *** Change this name to match desired region

        - Name: CreateChangeSet-region2
          Region: region2
          ActionTypeId:
            Category: Deploy
            Owner: AWS
            Version: 1
            Provider: CloudFormation
          InputArtifacts:
            - Name: BuildOutput
          Configuration:
            ChangeSetName: project-name-stack
            ActionMode: CHANGE_SET_REPLACE
            StackName: !Sub '${ProjectName}-stack'
            Capabilities: CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND
            TemplatePath: BuildOutput::template-export-region2.yml
            RoleArn: !Ref DeployRole
          RunOrder: 1
          RoleArn: !Ref PipelineRole

    - Name: DeployToRegion2
      # *** Change this for desired region
      Region: region2
      ActionTypeId:
        Category: Deploy
        Owner: AWS
        Provider: CloudFormation
        Version: '1'
      InputArtifacts:
        - Name: BuildOutput
      Configuration:
        ActionMode: CREATE_UPDATE
        Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND
        RoleArn: !GetAtt CloudformationRole.Arn
        StackName: !Ref ApplicationStackName
        # *** Change this to match the region of this stack, 
        # *** using the file generated in the build step
        TemplatePath: BuildOutput::template-export-region2.yml
      RunOrder: 2

So as first Action here is creating changeset. This is very useful for testing it and to see what kind of changes are going to be deployed. You can put approval action too. As InputArtifacts we have the BuildOutput from the Build Action provided. Two main variables in this section is the “Region: region” and “RunOrder” (A positive integer that indicates the run order within the stage.) Also the TemplatePath is the template that was defined in the buildspec file. This can be the same template or different one depending of what you are trying to do.

  ArtifactStores: 
    -
      Region: region1
      ArtifactStore:
        Type: "S3"
        Location: !Ref S3BUCKETregion1
        EncryptionKey:
          Id: !Ref KMSKEYregion1
          Type: KMS
    -
      Region: region2
      ArtifactStore:
        Type: "S3"
        Location: !Ref S3BUCKETregion2
        EncryptionKey:
          Id: !Ref KMSKEYregion2
          Type: KMS

At the end we end up with defining the artifact or the artifact store for each region (with their corresponding kms keys). This should be the end results image of the pipeline

Note: There should be IAM Roles and Assume Role for the AWS Account (Tools) to have access to other environments and Roles for the pipelines to execute the code.

Source: Cross Region deployments with AWS CodePipeline Cross-Region Actions with CodePipeline on AWS Thanks to @mikebroberts for some tips.

GitHub Profile Page

Sun, 30 Aug 2020 00:00:00 +0000

Building your network and sharing your work are probably the best ways to get your names out there in the community.

As some of you may already know, GitHub recently added a hidden feature that lets you customize your GitHub profile page.

And here is how to do it. Once you log into your GitHub account, create a new repository and name it as your Username. Also, make it Public and check the “Initialize this repository with a README” file button. With this step you will get some default README template, and its very good for starting point.

After that you pin gists and repositories to your profile so other people can quickly see your best work.

You can add even list of some YouTube channels or Blogs from pages. And in order to do this you need to create some Action workflows in the github. Here are the steps to do this:

Create a folder named .github and create workflows folder inside of it.
Create a new file named in the workflowws directory, named youtube-workflow.yml with the following contents inside.

Note: Not forget to change the CHANNEL-ID with your channel id from YouTube.

For more details on the Actions Workflows you can check the gautamkrishnar or codeSTACKr GitHub.

name: Latest YouTube Videos
on:
  schedule:
    # Runs every hour
    - cron: '0 * * * *'
  workflow_dispatch:

jobs:
  update-readme-with-youtube:
    name: Update this repo's README with latest videos from YouTube
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: gautamkrishnar/blog-post-workflow@master
        with:
          comment_tag_name: "YOUTUBE"
          feed_list: "https://www.youtube.com/feeds/videos.xml?channel_id=<CHANNEL-ID>"

There are different feed list that you can add, like from “Stackoverflow”, “Spotify” or “Medium”.

Artificial Intelligence in In Vitro Fertilization

Fri, 20 Dec 2019 00:00:00 +0000

EMBRY is an application that follows the whole IVF Process and predicts the outcome.

In vitro fertilisation is a process of fertilisation where an egg is combined with sperm outside the body, in vitro. The process involves monitoring and stimulating womans ovary and removing oocyte and letting sperm fertilize them in a liquid, in a laboratory.

As we all know AI helps a lot in different fields, especially in medicine. So this is just another field where we try to implement this AI concept and be of help to embriologists as another technology that they can use in order to increase the success rate in IVF.

How it starts:

We start with a simple experiment, trying do detect mature eggs. Together with my wife, Mimoza Adji - Krsteva, who is an embryologist and a guide in IVF Procedure in this project, we spent some time (a lot), on more than 6500 images of eggs, trying to train the model to detect the quality. After some time we did the same for embrios, trying to detect in which day they are. Now we are working on detecting not just the quality but other details too, on the eggs and the embrios. The same methods will be applied on the sperm too. One thing to another we develop this concept and app that can predict the IVF outcome. You know how hard it can get when you work with someone who is always right 😃

What is EMBRY:

EMBRY is an application that follows the whole IVF Process and predicts the outcome using AI concept. Embriologists write information about the patients and follow the whole cycle. Based on the information and patient history data we can calculate the outcome. On the other hand using the camera from the microscope, we developed another tool called CEMBRY, option for detection, that can help embriologists to faster detect the quality and any anomaly if they have.

We start with detection of mature eggs (The first picture)

CEMBRY can detect mature eggs, detect how old are the embrios, and currently we are finishing the process on detecting not just the quality of the embrios but other details, which currently are written manually by the embriologists.

As I mentioned above, all this information is written and taken in calculation. Prediction is the outcome of the algorithm calculation, based on the information from the patients in percentages. Beside the prediction part, we are opening another door for research. Hospitals, Clinics and Research centers can analyze patients data and try to find other problems for solving.

Detection of the day on Embrios

Benefits

With this kind of method and technology, we increase the success rate of IVF, the speed, and embriologists are more accurate in their choosing. Saving the materials for further attempts if needed. Research centers even hospitals and clinics can participate in solving new problems. They can use this tool for educational purpose too. Using this kind of technique we can save money that the goverment is giving for the first tree attempts and money that the patient is giving … The next phases are detection of morphology, getting details, and suming the quality of the eggs and embrios, which can later be applied to the sperm too.

The whole application is written in Python and Django as a framework. The hospital/clinics will have overview of the patients and results. No private data is stored, like Name and surnames, just results and numbers. Each hospital/clinic will have their own ID for each patient. They can re-calculate each attempt if they need to.

Any IVF Hospital or clinic that wants to have or be part of this interesting research and prediction concept, can feel free to e-mail me. For any other details you can fallow us on Facebook and Slack.

Some overview of the app

Steps to create Kubernetes Cluster Federation

Thu, 31 Oct 2019 00:00:00 +0000

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.

Kubernetes Cluster Federation, which was first released in version 1.3 back in July 2016, allows you to federate multiple clusters together and then control them as a single entity using a Federation control plane. Federation supports adding clusters located in different regions within the same cloud provider network, clusters spanning across multiple cloud providers and can even include on-premise clusters. This can be achieve by providing 2 major building blocks.

So here are some reasons why to use federation

Sync resources across clusters: Federation provides the ability to keep resources in multiple clusters in sync. For example, you can ensure that the same deployment exists in multiple clusters.
Cross cluster discovery: Federation provides the ability to auto-configure DNS servers and load balancers with backends from all clusters. For example, you can ensure that a global VIP or DNS record can be used to access backends from multiple clusters.

Some other use cases that federation enables are:

High Availability
Avoiding provider lock-in Federation is not helpful unless you have multiple clusters.

Some of the reasons why you might want multiple clusters are:

Low latency
Fault isolation
Scalability
Hybrid cloud

This scripts, that you can find on my Git, help you create Kubernetes Cluster on AWS in different regions and then create Federation between clusters.

Implementation

First you need to configure the AWS Credentials or AWS Variable in the shell where you will execute the scripts. This can be done using the AWS CLI. If you don’t have it, you can install it using pip (“pip install awscli –upgrade”).

For general use, the aws configure command is the fastest way to set up your AWS CLI installation.

$ aws configure
AWS Access Key ID [None]: AWSACCESSKEYID
AWS Secret Access Key [None]: ******************************
Default region name [None]: us-west-2
Default output format [None]: json

Then start execute the scripts for creating the Clusters in regions. example: K8CUSA.sh will create Kubernetes Cluster in USA ‘us-east-1’ Region. You must have minimum 2 Clusters in order to Join them in Federations. You can find the other two scripts for creating Cluster in Japan and Europe (K8CJPN.sh and K8CEUR.sh).

After You create the cluster you can then execute the K8CJoin.sh in order to join them in cluster. If you have other cluster that you want to be in the federation you will have to add them in the script. To do this you have to add at the botom of the script another line:

kubefed join ${ANOTHER_CLUSTER_ALIAS} --host-cluster-context=${USA_CLUSTER_ALIAS} --cluster-context=${ANOTHER_CLUSTER_ALIAS}

The host cluster context is the host cluster hosting the components that make up the Federation control plane.

Note: This is old way but usefull in order to start. Its possible that some images during the creation of cluster would not exists or are out of date, but that part is easy to fix.

To delete the cluster, use the script that start with “DEL”, (DELK8CUSA.sh, in order to delete USA Cluster)

code snippet test

Thu, 21 Dec 2017 00:00:00 +0000

This is a raw snippet:

hello world
123
This is a text snippet

This is a JavaScript snippet:

const add = (a, b) => a + b
const minus = (a, b) => a - b

console.log(add(100,200))  // 300
console.log(minus(100,200))  // -100

This is a Python snippet:

def say_hello():
    print("hello world!")

say_hello()   // "hello world!"

Side note comment: applied a bug fix similar to this commit to ensure code snippet does not collapse unexpectly upon clicking on it. This issue is tracked here.

3 Steps (2 minutes) to Setup Your Personal Website with Jalpc

Tue, 31 Jan 2017 00:00:00 +0000

Everyone wants to have a personal website, you can display your infomation to public, post blogs and make friends. If you are CS engineer, haveing a self website will benefit your interview.

So, if you like this website https://jarrekk.github.io/Jalpc/ or http://www.jarrekk.com and are willing to have a website, here is a way to build your website in 3 steps(2 minutes). Following are steps to setup your website(make sure you have basic knowledge of Jekyll and GitHub Pages, if you want to custom css/js NPM is needed):

Fork this project – Jalpc at GitHub. If you want to edit website at github, do it as following gif or clone forked repository. git clone git@github.com:github_username/Jalpc.git.
Enter into repository directory and edit following file list:
- _config.yml: edit ‘Website settings’, ‘author’, ‘comment’ and ‘analytics’ items.
- _data/landing.yml: custom sections of index page.
- _data/index/: edit sections’ data to yours at index page, please notice comment at each file.
- _data/blog.yml: edit navbar(categories) of blog page, if you have different/more blog page, copy blog/python.html and change it to your category HTML file, and edit Python, /python/ to your category name at items title and permalink, make sure title is the same as permalink but capitalized first letter(except HTML).
- CNAME: If you wanna release website at your own domain name: edit it and create gh-pages branch; if you want to use github_username.github.io: leave it blank.
- Go to repo’s settings panel, config GitHub Pages section to make sure website is released.
Push changes to your github repository and view your website, done!

From now on, you can post your blog to this website by creating md files at post/ directory and push it to GitHub, you can clear files at this directory before you post blogs.

If you like this repository, I appreciate you star this repository. Please don’t hesitate to mail me or post issues on GitHub if you have any questions. Hope you have a happy blog time!😊