Delphix Toolkits (dxToolkit and dxmToolkit)

 View Only
Expand all | Collapse all

Help with dxToolkit on RHEL

  • 1.  Help with dxToolkit on RHEL

    Posted 03-20-2024 12:46:00 PM
    Edited by Michael Torok 03-20-2024 01:07:56 PM

    Created to move a subthread to a new post.
    Please see below.



    ------------------------------
    Michael Torok
    Digital Customer Experience, Senior Director
    Delphix
    ------------------------------



  • 2.  RE: Help with dxToolkit on RHEL

    Posted 03-08-2024 11:36:00 AM

    Hi Michael,

    I'd like to use this thread to raise the question for the same subject.

    I have installed dxToolkit many times in different RedHat machine and running fine. However the recent installation on the new ReadHat machine gave me some trouble.

    I configure the dxtools.conf as I usually did, then I try to test with various ./dx_xx_xx command. But nothing happened - no error, no result returns.

    i.e. 

    --$ ./dx_get_cpu
    --$

    --$ ./dx_get_env

    --$



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 3.  RE: Help with dxToolkit on RHEL

    Posted 03-08-2024 11:44:00 AM

    Hello,

    It is generally best to start new threads for new questions because it helps others find answers when they have the same questions.

    Please provide the following output:

    1. Run "id -a" as the user trying to execute the dxtoolkit commands.
    2. Change into the "/tmp" directory and run "ls -la | more" and provide the output of the first two lines (. and .. directory permissions).
    3. From the "/tmp" directory, run "ls -la | grep par". Dxtoolkit creates a directory in "/tmp" with the name par and a number (par-737962617365 for example). If execute permissions are somehow removed from the "/tmp" directory, it can cause the dxtoolkit commands to execute with no result.

    Thanks,

    Neal



    ------------------------------
    Neal Stack
    Senior Principal Technical Support Engineer
    Delphix
    ------------------------------



  • 4.  RE: Help with dxToolkit on RHEL

    Posted 03-08-2024 02:22:00 PM

    Thank you Neal for the quick response. Here are the output

    --(~)-$ id -a
    uid=800(oracle) gid=26013(oinstall) groups=26013(oinstall),26629(oradba),400005628(rpa) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

    --(/tmp)-$ ls -la | more
    total 9940
    drwxrwxrwt. 18 root           root                460 Mar  8 20:15 .
    dr-xr-xr-x. 22 root           root               4096 Jan 22 12:57 ..

    --(/tmp)-$  ls -la | grep par
    drwx------.  4 oracle         oinstall             80 Mar  8 16:55 par-6f7261636c65

    I also run step 2 and step 3 on another server which has dxToolkit running well, the output are identical with the one from the server we have dxtoolkit issue, here are the output from "good" server:

    --(/tmp)-$ ls -al | more
    total 7088
    drwxrwxrwt. 4196 root           root           262144 Mar  8 21:19 .
    dr-xr-xr-x.   29 root           root             4096 Mar  3 09:12 ..

    --(/tmp)-$ ls -l | grep par
    drwx------. 3 oracle         oinstall          60 Mar  8 18:05 par-6f7261636c65



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 5.  RE: Help with dxToolkit on RHEL

    Posted 03-08-2024 02:37:00 PM

    Hello,

    I can't spot a permission problem but there are a few things you can try...

    1) You can move the old directory out of the way and try to run the command again. If you "mv par-6f7261636c65 par-6f7261636c65_BAD" and execute the command again, dxtoolkit will build a new directory as needed.

    2) If that doesn't help, you can run "strace -f -o /tmp/dx_strace.out ./dx_get_cpu" and review the strace output for errors.

    If you are unable to spot the problem in the strace output, I've reached out to your Customer Success Team so they may try and get in touch with you for an interactive debugging session.

    Thanks,

    Neal



    ------------------------------
    Neal Stack
    Senior Principal Technical Support Engineer
    Delphix
    ------------------------------



  • 6.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 02:08:00 PM

    I move the old "par" directory to par-xxxx.bad and run "strace -f -o /tmp/dx_strace.out ./dx_get_cpu" again.

    Yes, a new /tmp/par-xxx dir is generated. And the last 3 lines in the trace file /tmp/dx_strace.out shows below;

     chmod("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/libperl.so.5.26.735839", 0750) = 0
     rename("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/libperl.so.5.26.735839", "/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/libperl.so.5.26") = 0
     execve("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/dx_get_cpu", ["/tmp/par-6f7261636c65/cache-2c6e"...], 0x556929a34730 /* 42 vars */) = -1 EACCES (Permission denied)

    Looks like it's still a permission issue, but I have not idea what's wrong.

    Thanks

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 7.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 02:44:00 PM

    Here is full trace file attached



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 8.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 02:59:00 PM

    Hi Johnley,

    Are you using the Upload file button to attach your file? I don't see a file. You might want to upload to the site Neal mentioned below, in any case: https://upload.delphix.com.

    Thanks,
    Michael


    ------------------------------
    Michael Torok
    Digital Customer Experience, Senior Director
    Delphix
    ------------------------------



  • 9.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 09:05:00 PM
      |   view attached

    Hi Michael,

    Actually I did, but just realized that file name .out is not supported for uploading; So just rename the file and upload again.

    Thank you

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------

    Attachment(s)

    txt
    dx_strace.txt   17 KB 1 version


  • 10.  RE: Help with dxToolkit on RHEL

    Posted 03-08-2024 02:44:00 PM

    Hello,

    If you need to upload the strace output, you can upload it securely to our upload site (https://upload.delphix.com). Only Delphix employees can download files from that site.

    Thanks,

    Neal



    ------------------------------
    Neal Stack
    Senior Principal Technical Support Engineer
    Delphix
    ------------------------------



  • 11.  RE: Help with dxToolkit on RHEL

    Posted 03-18-2024 02:40:00 PM

    Hi Neal, any progress or update? I did upload the full trace last week. Now all my work is stuck with the dxToolkit setup.

    Thank you

    Regards

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 12.  RE: Help with dxToolkit on RHEL

    Posted 03-18-2024 03:49:00 PM

    Hello Johnley,

    It looks like Technical Account Manager Jose has asked some additional questions.

    I can't see anything from the trace file other than permission denied:

    741598 read(3, "2c6e277bc1db67f10f9dbf3d85961277"..., 41) = 41
    741598 mkdir("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae", 0700) = -1 EEXIST (File exists)
    741598 stat("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/dx_get_cpu", {st_mode=S_IFREG|0741, st_size=166608, ...}) = 0
    741598 stat("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/libperl.so.5.26", {st_mode=S_IFREG|0750, st_size=2198216, ...}) = 0
    741598 execve("/tmp/par-6f7261636c65/cache-2c6e277bc1db67f10f9dbf3d859612778ef2fcae/dx_get_cpu", ["/tmp/par-6f7261636c65/cache-2c6e"...], 0x555a01fc3730 /* 42 vars */) = -1 EACCES (Permission denied)



    ------------------------------
    Neal Stack
    Senior Principal Technical Support Engineer
    Delphix
    ------------------------------



  • 13.  RE: Help with dxToolkit on RHEL

    Posted 03-11-2024 03:37:00 AM

    Hi Johnley.
    I've seen this behaviour in other brand new installed RHEL hosts. I haven't been able to fully root cause it but this seems to be related to Selinux.

    Execute the command getenforce both on the "good" and the "bad" servers. If only the "bad" one has Selinux enabled, try with setenforce 0 to temporarily disable Selinux and try using dxToolkit.

    If both systems have Selinux enabled, you will have to modify the context of the dxToolkit files/user to be able to write to /tmp. This is a RHEL article that may help explain this: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/using_selinux/index

    This is only a hypothesis at this moment, but seems to be the most feasible.



    ------------------------------
    Jose Manuel Rodriguez Prieto
    Technical Account Manager
    Delphix
    ------------------------------



  • 14.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 12:48:00 PM

    Thank you so much Jose, you are my guru. It perfect resolved my problem, save me huge time in this tricky issue.

    Appreciate for your great help!

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 15.  RE: Help with dxToolkit on RHEL

    Posted 03-14-2024 01:55:00 PM

    Sorry, celebrated too early :-(. I mistakenly did the testing/investigation on a wrong server which was actually never had any issue with dxToolkit even with SELinux "enforcing"; 

    I tried the solution you provided on the "bad" server, it doesn't help. I still get the same issue -- dx_xxx command doesn't return error nor result even the SELinux is disabled.



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 16.  RE: Help with dxToolkit on RHEL

    Posted 03-18-2024 04:58:00 AM

    Hi Johnley.
    Can you execute dxtoolkit with strace and provide the output?
    Also, what is the exact version of RHEL in use?

    Can you try executing dxtoolkit with root privileges and let us know what happens?



    ------------------------------
    Jose Manuel Rodriguez Prieto
    Technical Account Manager
    Delphix
    ------------------------------



  • 17.  RE: Help with dxToolkit on RHEL

    Posted 03-19-2024 09:56:00 AM

    Hi Jose,

    The trace has been uploaded to the thread with Neal and Michael, but I just uploaded again just now through https://upload.delphix.com.

    My RHEL version is "Red Hat Enterprise Linux release 8.9 (Ootpa)"

    I run the ./dx_get_cpu from the root account, same symptom -- No error, no result returns, the command just returns the control to the OS.

    Thank you

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 18.  RE: Help with dxToolkit on RHEL

    Posted 03-19-2024 03:59:00 PM

    Hi Jose, 

    I resolved the issue and found the root cause. my /tmp is mounted as "tmpfs" which is a FS type saving everything in memory instead of physical disk, that is the root cause. I created a new volume on disk and remount /tmp with the new logical volume, dxToolkit just works fine then.

    Looks like dxToolkit doesn't support that FS type

    Thanks for your diligent response and follow up.

    Regards

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 19.  RE: Help with dxToolkit on RHEL
    Best Answer

    Posted 03-20-2024 04:19:00 AM

    Hello Johnley,

    I think the main problem is if /tmp file system is mounted with noexec flag set.

    dxtoolkit is using temporary location to extract Perl executable and then to execute it.

    You can define a temporary location by setting those 2 environment variables:



    ------------------------------
    Marcin Przepiorowski
    Director, Technical Services
    Delphix
    ------------------------------



  • 20.  RE: Help with dxToolkit on RHEL

    Posted 03-20-2024 11:08:00 AM

    Thank you, yes that's correct. Jose replied my thread with the same answer. The problem was resolved after I remounted /tmp without "noexec" option. Of course your workaround would work too although we didn't try.

    Thank you again for the knowledge sharing.

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------



  • 21.  RE: Help with dxToolkit on RHEL

    Posted 03-20-2024 04:22:00 AM

    Hi Johnley.

    I am glad you found a workaround to fix your problem.

    Based on your tips I found out that /tmp is mounted with the noexec option in RHEL8 which prevents the execution of binaries stored in /tmp.

    That explains why mounting /tmp on a different disk with the default options that do not include noexec fixes the issue.
    I just found out that you can define the variables TEMP or TMP in your dxToolkit shell environment to point them somewhere outside /tmp, like /home/delphix/temp or similar. This should fix the issue as well without reducing the security of your Linux box.



    ------------------------------
    Jose Manuel Rodriguez Prieto
    Technical Account Manager
    Delphix
    ------------------------------



  • 22.  RE: Help with dxToolkit on RHEL

    Posted 03-20-2024 11:11:00 AM

    Thank you Jose for the knowledge sharing which perfectly corrected my thought. I thought the root cause is the type of FS mount -- 'tmpfs'; Your sharing makes more sense that mount option of 'noexec' is the root cause.

    Thank you

    Regards

    Johnley



    ------------------------------
    Johnley Li
    Intern
    HSBC Canada
    ------------------------------