Delphix Products

Expand all | Collapse all

Working onFree trail Delphix on AWS.

Jump to Best Answer

Harsha -05-03-2017 02:34:00 PM

Adam Bowen05-05-2017 11:33:00 AMBest Answer

  • 1.  Working onFree trail Delphix on AWS.

    Posted 05-03-2017 04:57:00 AM
    Hi, I'm working on installing delphix engine, source and target environments into AWS individually by editing main.tf file.
    Step 1: I deleted all the source and target environment related stuff from main.tf file and let it create a new VPC, subnet, Internet Gateway and route tables.
    I was successful in creating all these. (What I observed is when created all 3 at once, there will be one VPC, one subnet, one route and one gateway created only).
    Step 2: I edited main.tf file again and gave only source related stuff and gave the subnet created in first step and tried. And when Im trying to apply the changes, the subnet which got created is getting destroyed and re-creating.

    I hope I made sense. Can someone help me on how to do this process. Im lost in the VPC, subnet things as Im not familiar with networking related things.  


  • 2.  RE: Working onFree trail Delphix on AWS.
    Best Answer

    Posted 05-03-2017 09:54:00 AM
    I wrote it, so I can help. What exactly are you trying to accomplish by separating things out?

    Are you executing everything from the same folder? Can you post your first and second TF?


  • 3.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 11:30:00 AM
    Hi, The thing is I'm going to have such a requirement where I need to create a delphix engine in one server and add a server to the same VPC and subnet. So I'm just trying like this now.

    my first tf is:
    #VERSION=017

    terraform {
        required_version = ">= 0.8.2"
    }

    # Specify the provider and access details
    provider "aws" {
      region = "${var.aws_region}"
      access_key = "${var.access_key}"
      secret_key = "${var.secret_key}"
    }

    data "aws_ami" "de_ami" {
      most_recent = true
      filter {
        name = "name"
        values = ["Delphix Engine 5.1.4.0 Free Trial"]
      }
      owners = ["180093685553"]
    }


    resource "aws_security_group" "landshark" {
      name = "${var.instance_name}-${aws_vpc.main.id}"
      description = "Allow all inbound traffic"
      vpc_id = "${aws_vpc.main.id}"
      ingress {
          from_port = 0
          to_port = 0
          protocol = "-1"
          cidr_blocks = ["10.0.1.0/24", "${var.your_ip}/32"]
      }

      egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
      }

      tags {
        Name = "${var.instance_name}-${var.image_base}-allow_all"
      }
    }

    resource "aws_instance" "de" {
      instance_type = "m4.xlarge"
      ami = "${data.aws_ami.de_ami.id}"

      key_name = "${var.key_name}"

      vpc_security_group_ids = ["${aws_security_group.landshark.id}"]

      subnet_id = "${element(aws_subnet.aw_sub.*.id, 0)}"
      private_ip = "10.0.1.10"
      
      #Instance tags
      tags {
        Name = "${var.instance_name}_${var.image_base}_DE"
      }
    }


    resource "aws_vpc" "main" {
        cidr_block = "10.0.1.0/24"
        enable_dns_hostnames = true
        tags {
            Name = "${var.instance_name}_${var.image_base}_vpc"
        }
    }

    resource "aws_route" "r"{
      route_table_id = "${aws_vpc.main.main_route_table_id}"
      destination_cidr_block = "0.0.0.0/0"
      gateway_id = "${aws_internet_gateway.main.id}"
    }

    resource "aws_internet_gateway" "main" {
        vpc_id = "${aws_vpc.main.id}"
        tags {
            Name = "${var.instance_name}_${var.image_base}_ig"
        }
    }

    resource "aws_subnet" "aw_sub" {
        vpc_id = "${aws_vpc.main.id}"
        cidr_block = "10.0.1.0/24"
        map_public_ip_on_launch = true
        tags {
            Name = "${var.instance_name}_${var.image_base}_sub}"
        }
    }

    output "DE" {
      value = "${
        formatlist(
          "Delphix Engine - Public IP: %s Private IP: %s\n    Access via http://%s\n    Username: delphix_admin Password: landshark",
          aws_instance.de.*.public_ip,
          aws_instance.de.*.private_ip,
          aws_instance.de.*.public_ip
          )}"
    }


    variables declaration from here --- im just leaving them

    my second tf is (As I have already created necessary one VPC, subnet, Internet Gateway and route
    #VERSION=017

    terraform {
        required_version = ">= 0.8.2"
    }

    # Specify the provider and access details
    provider "aws" {
      region = "${var.aws_region}"
      access_key = "${var.access_key}"
      secret_key = "${var.secret_key}"
    }

    data "aws_ami" "ls_ami" {
      most_recent = true
      filter {
        name = "name"
        values = ["Delphix Oracle 11G Linux Source"]
      }
      owners = ["180093685553"]
    }

    resource "aws_security_group" "default" {
      vpc_id = "${var.vpc_id}"
      ingress {
          from_port = 0
          to_port = 0
          protocol = "-1"
          cidr_blocks = ["10.0.1.0/24", "${var.your_ip}/32"]
      }

      egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
      }

     }

    resource "aws_instance" "ls" {
      instance_type = "m4.large"
      # Lookup the correct AMI based on the region
      # we specified
      ami = "${data.aws_ami.ls_ami.id}"
      connection {
        type = "ssh"
        user = "centos"
        private_key = "${file("${var.key_name}.pem")}"
        timeout = "10m"
      }


      key_name = "${var.key_name}"

      # Our Security group to allow HTTP and SSH access
      vpc_security_group_ids = ["${aws_security_group.default.id}"]

      subnet_id = "${var.subnet_id}"
      private_ip = "10.0.1.20"
      depends_on = ["aws_internet_gateway.main"]

      user_data = "service dbora start"
      provisioner "remote-exec" {
        inline = [
        "sudo cp /usr/bin/tput /usr/bin/tput.bak; sudo cp /dev/null /usr/bin/tput"
        ]
      }

      provisioner "remote-exec" {
        inline = [
        "sudo sed -i -e 's|^MODULE_BASE=.*|MODULE_BASE=GA|' /home/delphix/.ls/config; sleep 60; sudo /u02/app/content/landshark_fetch free_trial -F y"
        ]
      }

      provisioner "remote-exec" {
        inline = [
        "sudo mv /usr/bin/tput.bak /usr/bin/tput"
        ]
      }

      #Instance tags
      tags {
        Name = "${var.instance_name}_${var.image_base}_LS"
      }

    }

    resource "aws_route" "r"{
      route_table_id = "rtb-51083135"
      destination_cidr_block = "0.0.0.0/0"
      gateway_id = "igw-3ca6a459"
    }


    output "LS" {
      value = "${
        formatlist(
          "Linux Source - Public IP: %s Private IP: %s\n    Access via SSH @%s\n    Username: delphix Password: delphix\n    Prod Employee App: http://%s:1080";,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.private_ip,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.public_ip
          )}"
    }


    variables declaration here ------ the below two are added by me and given the vpc and subnet details in terraform.tfvars file
    variable "vpc_id" {}
    variable "subnet_id" {}


    Either the internet gateway is modifying itself and destroying the instaces associated with it. In this case, the DE which I created in the first step.

    I almost got succeeded at one point, but an error with aws_internet_gateway popped up and ruined everything. 


  • 4.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 11:42:00 AM
    Ok. So, a couple of things:

    1. Do everything from the same folder, this makes things simpler
    2. Don't declare the same variables twice. 
    So, keep your first tf the same, just call it main.tf. 'terraform apply' to build it.
    Then, create a second tf in the same folder, say "ls.tf", and have it look like the below. You can then run 'terraform plan' to check out what terraform is going to do. I think it should say "add 1, delete 0, modify 0"

    resource "aws_instance" "ls" {  instance_type = "m4.large"
      # Lookup the correct AMI based on the region
      # we specified
      ami = "${data.aws_ami.ls_ami.id}"
      connection {
        type = "ssh"
        user = "centos"
        private_key = "${file("${var.key_name}.pem")}"
        timeout = "10m"
      }
      # The name of our SSH keypair you've created and downloaded
      # from the AWS console.
      #
      # https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:
      #
      key_name = "${var.key_name}"
      # Our Security group to allow HTTP and SSH access
      vpc_security_group_ids = ["${aws_security_group.landshark.id}"]
      subnet_id = "${element(aws_subnet.aw_sub.*.id, 0)}"
      private_ip = "10.0.1.20"
      user_data = "service dbora start"
      # We run a remote provisioner on the instance after creating it.
      provisioner "remote-exec" {
        #This provisioner is to workaround the below issue:
        #https://github.com/hashicorp/terraform/issues/11091
        inline = [
        "sudo cp /usr/bin/tput /usr/bin/tput.bak; sudo cp /dev/null /usr/bin/tput"
        ]
      }
      provisioner "remote-exec" {
        inline = [
        "sudo sed -i -e 's|^MODULE_BASE=.*|MODULE_BASE=GA|' /home/delphix/.ls/config; sleep 60; sudo /u02/app/content/landshark_fetch free_trial -F y"
        ]
      }
      provisioner "remote-exec" {
        #This provisioner is to undo the workaround
        inline = [
        "sudo mv /usr/bin/tput.bak /usr/bin/tput"
        ]
      }
      #Instance tags
      tags {
        Name = "${var.instance_name}_${var.image_base}_LS"
      }
    }
    output "LS" {  value = "${
        formatlist(
          "Linux Source - Public IP: %s Private IP: %s\n    Access via SSH @%s\n    Username: delphix Password: delphix\n    Prod Employee App: http://%s:1080";,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.private_ip,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.public_ip
          )}"
    }


  • 5.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 12:19:00 PM
    Thanks for the quick reply, Adam. 

    I'm doing it from the same folder. So my final ls.tf is like this.
    #VERSION=017

    terraform {
        required_version = ">= 0.8.2"
    }

    # Specify the provider and access details
    provider "aws" {
      region = "${var.aws_region}"
      access_key = "${var.access_key}"
      secret_key = "${var.secret_key}"
    }

    data "aws_ami" "ls_ami" {
      most_recent = true
      filter {
        name = "name"
        values = ["Delphix Oracle 11G Linux Source"]
      }
      owners = ["180093685553"]
    }

    resource "aws_instance" "ls" {  instance_type = "m4.large"
      # Lookup the correct AMI based on the region
      # we specified
      ami = "${data.aws_ami.ls_ami.id}"
      connection {
        type = "ssh"
        user = "centos"
        private_key = "${file("${var.key_name}.pem")}"
        timeout = "10m"
      }
      # The name of our SSH keypair you've created and downloaded
      # from the AWS console.
      #
      # https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:
      #
      key_name = "${var.key_name}"
      # Our Security group to allow HTTP and SSH access
      vpc_security_group_ids = ["sg-b44117d3"]
      subnet_id = "${var.subnet_id}"
      private_ip = "10.0.1.20"
      user_data = "service dbora start"
      # We run a remote provisioner on the instance after creating it.
      provisioner "remote-exec" {
        #This provisioner is to workaround the below issue:
        #https://github.com/hashicorp/terraform/issues/11091
        inline = [
        "sudo cp /usr/bin/tput /usr/bin/tput.bak; sudo cp /dev/null /usr/bin/tput"
        ]
      }
      provisioner "remote-exec" {
        inline = [
        "sudo sed -i -e 's|^MODULE_BASE=.*|MODULE_BASE=GA|' /home/delphix/.ls/config; sleep 60; sudo /u02/app/content/landshark_fetch free_trial -F y"
        ]
      }
      provisioner "remote-exec" {
        #This provisioner is to undo the workaround
        inline = [
        "sudo mv /usr/bin/tput.bak /usr/bin/tput"
        ]
      }
      #Instance tags
      tags {
        Name = "${var.instance_name}_${var.image_base}_LS"
      }
    }

    output "LS" {
      value = "${
        formatlist(
          "Linux Source - Public IP: %s Private IP: %s\n    Access via SSH @%s\n    Username: delphix Password: delphix\n    Prod Employee App: http://%s:1080";,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.private_ip,
          aws_instance.ls.*.public_ip,
          aws_instance.ls.*.public_ip
          )}"
    }

    variable "access_key" {
      description = "Amazon AWS Access Key"
    }
    variable "secret_key" {
      description = "Amazon AWS Secret Key"
    }

    variable "image_base" {
      default = "Delphix Free Trial with Oracle 11G"
    }

    variable "aws_region" {
      description = "The aws region where you will deploy."
    }

    variable "your_ip" {
      description = "Your IP address (for restricting access to your environment)."
    }

    variable "key_name" {
      description = "The name of the AWS Key Pair you will use with this environment."
    }

    variable "instance_name" {
      description = "Any word to help identify your instances in AWS."
    }

    variable "community_username" {
      description = "Your Delphix Community username."
    }

    variable "community_password" {
      description = "Your Delphix Community password."
    }

    variable "subnet_id"
    {}

    After I run the terraform plan , I got like this --> 
    Plan: 1 to add, 0 to change, 6 to destroy.

    Everything which I created before is trying to delete. Im not sure why is that. It's just keep on destroying my previous things and creating new ones. 

    Im sorry to drag like this.



  • 6.  RE: Working onFree trail Delphix on AWS.
    Best Answer

    Posted 05-03-2017 12:56:00 PM
    you ignored all of my advice. :-) Copy and paste the blueprint I provided you in my last post, and try that. the. main.tf and ls.tf should both be present in the folder


  • 7.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 12:58:00 PM
    you might want to start with the tutorials on the terraform website, as they give you simpler blueprints to work with for learning this basic functionality


  • 8.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 02:34:00 PM
    thanks Adam, I understood now :) The instance got created. You made my day.
    Now The LS got attached directly as a dSource in the delphix engine, but its saying inactive. I'm able to access the LS URL and did some deletes and it was working.

    I'll try to re-create whole steps and watch. Thanks for the help anyways. :)  


  • 9.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 02:34:00 PM
    ya I should do it., I know :)


  • 10.  RE: Working onFree trail Delphix on AWS.

    Posted 05-03-2017 02:54:00 PM
    Yes, the dSource is inactive. When the LT is provisioned, it executes the automation to enable the items in the Delphix engine. You can enable the dSource manually, though.


  • 11.  RE: Working onFree trail Delphix on AWS.
    Best Answer

    Posted 05-04-2017 10:01:00 AM
    Glad everything is working for you. I hope you are enjoying the free trial.


  • 12.  RE: Working onFree trail Delphix on AWS.

    Posted 05-04-2017 11:34:00 AM
    Hi Adam, Good Morning. One last question, pls. When I'm trying to use the existing VPC and Subnet for launching a DE instance alone, it is deleting all the other previous instances which are with that VPC. I'm giving the details in vpc_security_group_ids = ["vpc- xxxxxx"] and subnet_id = "subnet-xxxxxx" . May I know why. 


  • 13.  RE: Working onFree trail Delphix on AWS.

    Posted 05-04-2017 12:24:00 PM
    It is hard to answer without more details....But, Terraform is about "immutable infrastructure." So, everytime you launch terraform it checks the blueprints against the data in the tfstate file. So, terraform created objects in your VPC in one set of blueprints, and then you replace those blueprints with a new set of blueprints that don't contain those objects, then terraform thinks you want to destroy them (since they are no longer in the plans). 

    Example:
    1. I just have a simple blueprint where terraform creates a subnet. 
    2. terraform apply creates the subnet with id "subnet-abc"
    3. I then replace that blueprint with a blueprint that builds a server and places it in a subnet names "subnet-abc"
    4. terraform apply notices the instruction to create subnet-abc that you had it build in the first blueprint is no longer a part of the plans, so it destroys it.
    Does this make sense?

    Terraform blueprints define the immutable state, as opposed to something like Chef or Puppet that will just build on top of whatever the current state is.


  • 14.  RE: Working onFree trail Delphix on AWS.

    Posted 05-05-2017 06:05:00 AM
    May be I wasn't clear, as you said. 

    You asked me to keep all *.tf files in the same folder. I was trying w/o putting everything.

    So my first main.tf file will create all the VPCs, SUBNETS, SGs and everything. And my second ls.tf (alone) should be created w/o main.tf in the same folder. So I just gave the security_group and subnet id in the aws_instance of ls.tf and is working as expected. I was trying this yesterday and couldn't get it. So I asked you. Now I'm ok as it is working.

    As you said, it checks the .tfstate file before it tries creating. So we might need to empty that before 'apply'. Then our output will be 
     1 to add, 0 to change, 0 to destroy. 

    One more thing , After creating ls.tf, you said, the dSource can be enabled from UI itself if the target is not installed. When I try to enable, it said, "Cannot connect to source" . 

    Anyways, thanks Adam for all the help.


  • 15.  RE: Working onFree trail Delphix on AWS.

    Posted 05-05-2017 06:18:00 AM
    Hi, What you described is the default behavior of terraform,to enforce it to recreate the ressources and so you have to run those commands in order. Terraform destroy Will clean generated files by the first run Terraform apply Will create again all files and so with out prob Regards, Mouhssine


  • 16.  RE: Working onFree trail Delphix on AWS.

    Posted 05-05-2017 10:04:00 AM
    A few things:
    1. Do not delete you tfstate file. Once you do that, terraform will no longer have any way of deleting/modifying that resource. For example, in the example you gave, terraform would not be able to destroy your VPC once you are done with it.
    2. The blueprints I gave you have automation that is coupled between the three VM's so that a lot if checks, etc are done and bring everything up into a running state. So, without looking into my code, I am not sure why the source couldn't enable (it might be the oracle listener is down). 
    3. I think I want to first redirect you back to terraform.io and go through their tutorials so that you can learn about some of the basics of how blueprinting and immutable infrastructure work. Then you would be able to easily add/delete individual pieces that you want to do. 
    My blue prints are always broken down into small pieces. My main.tf has just the basics, like aws credentials and regions. I have a variables.tf that holds my variables, a firewall.tf, networking.tf, outputs.tf, etc.

    If you commit to going through their tutorial first, I'll be glad to answer additional questions for you.

    https://www.terraform.io/intro/index.html


  • 17.  RE: Working onFree trail Delphix on AWS.

    Posted 05-05-2017 10:44:00 AM
    I observed that. if the tfstate file is empty, things can't be destroyed. But how do I re-use the components like vpc, subnet etc.,? I want them like below w/o destroying the previous instances. I have a requirement, where I need to add DE in a VPC, where the VPC is being used by 10-12 instances. The first time if I add DE to that VPC, it will be added w/o any issues. But what if I need to add one more instance to the same VPC using terraform? It checks the tfstate and gives me a destroying plan.

    resource "aws_instance" "lt" {
      instance_type = "m4.large"
      ami = "${data.aws_ami.lt_ami.id}"
      connection {
        type = "ssh"
        user = "centos"
        private_key = "${file("${var.key_name}.pem")}"
        timeout = "10m"
      }
       key_name = "${var.key_name}"
       vpc_security_group_ids = ["sg-xxxxxxxx"]
       subnet_id = "subnet-xxxxxxxx" 
      private_ip = "10.0.1.30"
      user_data = "service dbora start"  
      provisioner "remote-exec" {
        inline = [
        "sudo cp /usr/bin/tput /usr/bin/tput.bak; sudo cp /dev/null /usr/bin/tput"
        ]
      }
      provisioner "remote-exec" {
        inline = [
        "sudo sed -i -e 's|^MODULE_BASE=.*|MODULE_BASE=GA|' /home/delphix/.ls/config; sleep 60; sudo /u02/app/content/landshark_fetch free_trial -F y"
        ]
      }

    I'm going through the terraform documents already. :) 


  • 18.  RE: Working onFree trail Delphix on AWS.
    Best Answer

    Posted 05-05-2017 11:17:00 AM
    Adam, I guess I understood now. 

    I was confused. If I want to add more instances using terraform, I can add more .tf files. 
    If I want to add the existing VPCs, all I need to do is add those values to the respective fields (to all the .tf files in the folder). Terraform can't destroy which it didn't create. So I'm safe even if I want to destroy :)


  • 19.  RE: Working onFree trail Delphix on AWS.
    Best Answer

    Posted 05-05-2017 11:33:00 AM
    Correct.