Linux, Red Hat, RHV, XenDesktop

Using Citrix PVS to stream Linux VDA (RHEL 7 Workstation)

I’ve been waiting a while to see this day and a sneaky section of the Citrix XenDesktop/PVS 7.12 notes indicate not only a lot more stability for the VDA Agent on RHEL but also that PVS supports (again) Linux targets!

If you’re not familiar with RHEL 7.x (7.3 in this case) you likely can use the same methods with CentOS 7 and Fedora.  If you don’t have a RHEL subscription, I would urge you to use CentOS 7, however, you can get a RHEL 7 subscription on the Developer portal (register here) (note it may just say 7.2 but once you run yum update and reboot it’ll be 7.3).

Let’s set the stage with what we are doing here.  We will be installing Red Hat Enterprise Linux (RHEL) Workstation 7.3 and configuring it to be a VDI Desktop in Citrix XenDesktop 7.12.  We are also using PVS 7.12 to stream a gold image to a VM that PXE boots and uses a cache disk (I choose to use cache to RAM overflow to disk).  Everything below is done as root for now. If you don’t like that, feel free to do it your own way but this is just a workstation we’re talking about here.

Install RHEL 7 Workstation

First thing you should do is install a VM, I’m using RHV4 and my VM consists of 2GB Memory, 1 vCPU and a 20GB disk.  The installation follows the guidelines in the documentation.  While their is an easy install option, we want to use the recommended best practice from Red Hat and leverage SSSD rather than samba.

First thing to notice is you need to scroll down to the section entitled “Prepare RHEL7/CentOS 7 for VDA Installation” which is not the first one (that is RHEL 6). Make sure you install Workstation and subscribe to get updates.  If you’re using RHEL you’ll need to run something similar to this

subscription-manager register
subscription-manager attach --pool={POOLID}
subscription-manager repos --disable=*
subscription-manager repos --enable=rhel-7-workstation-rpms \
                           --enable=rhel-7-workstation-extras-rpms \
                           --enable=rhel-7-workstation-optional-rpms \
yum clean all
yum install screen wget vim bash-completion git
yum update -y
#reboot if kernel is updated
systemctl reboot
#for ovirt/rhv I will install guest tools
yum install -y rhev-guest-agent
#if there is a bridge reported you might need to remove it before proceeding
#this would show up as some 192.x.x.x address in addition to your desired IP
#that wont play well with PVS and the VDA generally
#if that happens you can remove them with
#virsh net-destroy default
#virsh net-undefine default
#ifconfig #shouldnt have a virbr0 anymore

Follow the guide for setting the hostname and check hostname and hostname -f return the right values.  The command below enforces the hostname immediately.

hostnamectl set-hostname myhostname

Now modify chrony.conf for your ntp servers. You can use the following if you wish to run this in bash. Replace the IP addresses with the correct ones. (make sure you scroll to the right as the lines are long if you’re copy/pasting here)

sed -i 's/server iburst/#server iburst/g' /etc/chrony.conf
sed -i 's/server iburst/#server iburst/g' /etc/chrony.conf
sed -i 's/server iburst/server iburst/g' /etc/chrony.conf
sed -i 's/server iburst/server iburst/g' /etc/chrony.conf
systemctl restart chronyd

For JAVA, this is strange but if you use the default RHEL7 install the docs won’t work, that’s because they don’t have a symlink where Citrix says there is.  You should have java already installed if you’re using RHEL. I would recommend installing 1.8.0-openjdk and don’t remove the 1.7.0 if you’re using RHEL 7.2+ or CentOS7.  The path you want is /etc/alternatives/java. Let’s set it and source it in .bashrc (this will now always be a variable when you log in).

echo "export JAVA_HOME=/etc/alternatives/java" >> ~/.bashrc
. ~/.bashrc

Installing postgres should be simple, you can initialize without fear before we seal/clone so let’s do that.

yum install postgresql-server postgresql-jdbc -y
postgresql-setup initdb

systemctl start postgresql
systemctl enable postgresql

At this point you are ready for stage 2, the AD prep.  If you’re using XS or ESXi read the appropriate sections in the document.  I use Red Hat Virtualization because I like to live dangerously (and they also are my employer).

Active Directory Join (well not really)

This part can actually be done entirely and then we can remove the VM from the domain and clear credentials.  This isn’t a bad thing and if you do this, you’ll just want to leave the domain and delete the /etc/krb5.keytab (plus a few other things).  However we are going to do everything but join the domain at this point and leave that for later.

I HIGHLY SUGGEST you take a snapshot at this point.  If you love repeating steps (like the ones above, then ignore me but more than likely you’re going to mistype something here and need to revert).

Winbind sucks, so skip that part. We’re going with the latest, greatest SSSD.  My colleague, Dmitri Pal, discusses the best practice here if you’re a diehard winbind junkie.

Let’s run authconfig but also, we can specify all our variables up front now so you can just copy and paste and get on with life.  You can download them from this github and modify. Just follow the commands below

git clone
cd citrixpvslinux
vim #edit the file changing the options
chown root:root
chmod 600
. ./

If you type export and hit enter you should see the variables loaded for you.  We only want them for configuration so we load them now by sourcing the file (the . command above) and we will again after we create the gold image.

Let’s run our authconfig command now and modify the /etc/samba/smb.conf. You can run the github script or follow along. We aren’t joining the domain just yet though.

authconfig --smbsecurity=ads --smbworkgroup=${ADJOIN_SHORT} --smbrealm=${ADJOIN_REALM} --krb5realm=${ADJOIN_REALM} --krb5kdc=${ADJOIN_DC} --update
sed -i '/#--authconfig--end-line--/a kerberos method = secrets and keytab' /etc/samba/smb.conf

We will now prep the image a bit more for the PVS section, here we are writing in a file using our variables. You should check the file to be sure it looks right

cat >/etc/sssd/sssd.conf <

Now we are going to create a service that will join the machine to AD at boot time. We have to store the password in that file so we secure it, like we did above, with only root access.  This means you MUST be root to run this and access the file. Since we are going to join the machine with the command I HIGHLY suggest a minimum privileged account for this.  This guy seems to have a good walkthrough if you need one.  I also suggest you make an OU for the machines and add it as I have (it’s a variable in the export).  If you don’t do this, you’ll need to make the variable “Computers” or modify the command to join.

I suggest you use the github script and not copy and paste below.


cat >/etc/systemd/system/ctxvdaoneshot.service < /usr/local/sbin/ctxvdaoneshot
chmod 700 /usr/local/sbin/ctxvdaoneshot

cat >>/usr/local/sbin/ctxvdaoneshot <

To use this we would actually run ‘systemctl enable ctxvdaoneshot.service’ but we’re not quite ready yet.

At this point we’re as far as we’re going to get on AD without actually joining the machine. So let’s fix some issues that are going to pop up (I didn’t see any notes on this but you likely need to make these changes, otherwise your users will get some warnings)

Fixing the Display Issues

Run the following to fix the popups that occur if someone has a printer or moves the screen.  Most guest tools leverage VNC and this will prompt some security warnings (you can hit cancel and ignore them) about the color management and one about proxy for updates.  These are annoying and not easily fixed.  So the steps below will fix these issues by ignoring those warnings for anyone in the Domain Users group. Github script here.

(Update: Citrix had a ctx article on this now –

sed -e '$aX-GNOME-Autostart-enabled=false' -e '/X-GNOME-Autostart-enabled/d' -i.bak /etc/xdg/autostart/gnome-software-service.desktop

sed -i 's/Exec=\/usr\/bin\/vmware-user-suid-wrapper/#Exec=\/usr\/bin\/vmware-user-suid-wrapper/g' /etc/xdg/autostart/vmware-user.desktop

cat >/etc/polkit-1/rules.d/02-allow-colord.rules <

If you have GRID cards then good for you.  I don’t so I’m not going over that part.  However, if someone were to donate some to me I’d be happy to add that part, straight up trade.

Install the VDA Agent

Ok now we need to install this agent.  You need to actually download the rpm and put it somewhere. I use a local http server but you can just put it in the root directory.  scp works great for this. If you’re on a mac the command is something like:

scp ~/Downloads/Citrix/XenDesktopVDA- root@linuxvdafqdn:/root
# prompts for password of root and it will copy
# lets also copy the PVS agent (it is on the ISO for PVS 7.12)
scp ~/Downloads/Citrix/pvs_RED_HAT_7.12.0_BETA5_9201_x86_64.rpm root@linuxvdafqdn:/root

You should now have two rpm files in your ~ or /root directory. I guess you could use Citrix’s commands in the doc but I don’t do that. I generally always use the following for RPMs I find.  You can also use this for an rpm on a http site (just replace the location with the web address).

# install prerequisites if not already installed (it should warn you if not)
yum install -y ImageMagick motif foomatic-filters
rpm -Uvh ~/XenDesktopVDA-
systemctl enable ctxhdx ctxvda

At this point we’ve completed the VDA setup without having joined the machine to the domain.  A reboot and snapshot is advisable at this point.

systemctl poweroff
# take a snapshot
# bring it back up
# source variables
cd ~/citrixpvslinux
. ./

PVS Install

You should have the rpm in the /root directory or somewhere. Let’s go ahead and install it along with a prerequisite.

yum install -y tdb-tools
rpm -Uvh ~/pvs_RED_HAT_7.12.0_BETA5_9201_x86_64.rpm

The following commands are in this github script but you can follow along. We need to set selinux to permissive. This kind of sucks but another day I’ll work through that.  What you don’t want to do is disable selinux, you should NEVER be doing that honestly unless you’re exploring exploits or somehow time travelled prior to Y2K (buy some stock while you’re there!). Anyway let’s get this done.  I’m assuming selinux contexts aren’t transcribed properly when we stream the OS disk.

# Set to permissive for the immediate timeframe
setenforce Permissive
# Set it to persist on reboot (if you dont do this it will revert to enforced)
sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux

We also want to add our CA Trust because PVS runs securely now and the linux target need to trust the issuer.  You could use the option for self-signed in the citrix docs, but I’m assuming you’re serious about this and have a CA somewhere you can spin up a cert for.  You can grab the cert from the CA by using this command (it’s also part of the script).

openssl s_client -showcerts -connect ${ADJOIN_CA}:443 /dev/null|openssl x509 -outform PEM > /etc/pki/ca-trust/anchors/${ADJOIN_CA}.pem
update-ca-trust extract

Now let’s add a Cache Disk for PVS to use, I set mine up for 10G and created it without rebooting or shutting down.  My main disk is /dev/vda and the new disk is /dev/vdb.  If you used my script from github at this point, just ensure the variables are right and run it or run the following.

#export TGTDEV=/dev/vdb
sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' << EOF | fdisk $TGTDEV
 o # clear the in memory partition table n # new partition p # primary partition 1 # partition number 1 # default - start at beginning of disk # default, extend partition to end of disk p # print the in-memory partition table w # write the partition table q # and we're done EOF
mkfs.xfs -L 'PVS_Cache' ${TGTDEV}1

After this it’s time to seal up and convert our VM to a PVS image.  You’ll want to remove the subscription-manager info also to clear that for use but you can leave the hostnames alone since PVS will change them for you.  The github script for this part is here.

Let’s cleanup yum and also enable the vdaoneshot script too.  We assume you have the collection, store name and all that info correctly inputted to the or export.

yum clean all
subscription-manager unregister
subscription-manager remove --all

systemctl enable ctxvdaoneshot

# Lets create this

echo "If something went wrong please run: systemctl disable ctxvdaoneshot"
echo "before rebooting...."
echo "If you succeeded, please clone this image and remove the OS disk"
echo "Then create a template, create the VMs and add them to the collection in PVS"
echo "DO NOT FORGOT to set the boot order to use the NIC first"
echo "You also should have options 66/67 set in DHCP (they are the same old options from years ago)"

At this point you should be creating your image.  You’ll want to clone, remove the OS disk and then make additional VMs based on that VM. I had to manually create the additional VMs in PVS, transcribing the MAC addresses but it works.  This method creates Desktop VDI machines so select that when you create the machine catalog and delivery groups in XenDesktop.


Once all is complete you should be able to see your images and launch a desktop!

EDIT (12/22/16): I wanted to also mention the setup on the PVS side and 😄 side.

I am a big fan of stream to RAM but the notes on the Citrix site are a bit cryptic on the setting.  I understand them to say that Linux uses the RAM for cache that is there (Linux manages this) so the cache maximum is just a reserve and you want this set low to 8MB which is what I did.  I also have a disk setup as overflow.  It is vital you name this PVS_Cache in the mkfs.xfs command. I didn’t try lvm because I didn’t think it would work, I suggest you stick to xfs or ext4.  I love btrfs but unless you know a LOT about that filesystem I wouldn’t use it for default (the benefit of btrfs is generally the ability to add/remove disks from RAID without impact, kind of like lvm but lower level).


On the XenDesktop side, you can use the shared mode but I never had a lot of luck with the desktop there (Citrix said it should work so I’ll try later). I used the VDI mode for now.


Citrix, Uncategorized

Citrix Synergy as CTP

Getting to attend Synergy 2016 is one of the perks that accompanies being a CTP.  You also have responsibilities that come with this but I’m finding the perks seem to outweigh them!

I joined Red Hat last week and I think I’m the only Red Hatter here (I get weird looks with my hat but that’s on purpose).  Citrix is heavily aligned with Microsoft, Microsoft announced Red Hat on Azure and there has always been interest in VDI with Linux.  That being said, I get to come to Synergy as more an outsider than ever before (instead of from the customer or partner side).

CTPs have quite a few meetings lined up when they come to Synergy that start 2 days earlier than the kickoff.  These meetings are no joke and I got plenty of warnings about pacing myself.  They started at 8am promptly and continued to about 7pm two days in a row with few breaks.  The topics are intense and interesting and it’s easy to get overwhelmed.

The meetings are generally with the Citrix product teams and we get to learn and give feedback to new releases, ideas, successes and failures.  Each team is different in terms of how they operate.  I can’t go into too much detail but Citrix is very interested in improving their feedback loops and ensuring that products developed have purpose instead of just throwing things over a wall and surprising people.  This is a good move as it allows a lot more trail and error, and a lot less of “what the f is this thing?” when something in announced.  This is definitely the primary interest I had with my CTP meetings, in addition to the awesome company!

This year I’m not speaking at a breakout but doing some other activities.  I’ll be at the Synergy Showdown #XAonAzure with my team, the Village Idiots, along with Esther, Jarian and Paul from the CTP team.  We will be taking on the wily dutch team but our secret weapon hails from the Netherlands (Esther) so we are a lock in to win 😉 !  I will also have a few match.geek sessions if you want to hear me blather on about whatever you want and I’m hosting the education tech chat tables (except for today, Tobias will be doing Tuesday’s due to the showdown).

Today was fun, I got to sit at the bloggers table and get super secret early access the the keynote.  Esther even got me a front row table seat and the view is spectacular.  If you don’t think a CTP has perks, you’re mistaken and this is one of the best perks I’ve ever had.

If you’re here and want to connect, let me know. You can look for a dude wearing a red hat (literally) and it’s likely me or DM on twitter or the Synergy app. Looking forward to the rest of the week and running into old friend and making new ones!

Citrix, PVS, Uncategorized

Citrix PVS and Managed Service Accounts gMSA

I’m a big fan of Managed Service Accounts because they are much more secure and aren’t easily exploited by human beings.  Basically, Active Directory controls the account with it being responsible for changing passwords.  While use of gMSA (group managed service accounts) is sometimes hit or miss, I didn’t find much on recent use with Citrix other than a vague “we support this” statement.

Carl Webster had a much older attempt with PVS (not sure whether he tried again or not) and I wanted to ensure that this worked on PVS 7.7 (just released).

You’ll need a couple of things

I would leverage a tool for creating and managing gMSA that I got here.
(note: for a quick guide on setting this up, I would look through Derek Seaman’s blog).


Add you PVS server to the list otherwise it won’t work. (I only have 1 PVS server right now, I’m in rebuilding mode…)


For SOAP, you’ll need to make this account a member of the local admins on the PVS server (when you add the account, make sure you select “service accounts” for objects.


For SQL, I am using 2014 with availability groups.  Check out Derek’s blog for a great walkthrough on this.

Your database should have been created already (use the dbscript.exe to manually create the database in PVS)

Grant the permissions needed to your gMSA on the SQL database (I create the account on both database servers just in case (when I test the failover))

Testing failover should work and you will also notice the services are runningpvsconsoleservicespvs

Citrix, XenServer

FusionIO and XenServer

Getting FusionIO cards and XenServer to work seem to be a bit tougher than I thought but it’s possible. I have a few cards, some old servers and a XenServer 6.5 ISO.  I was hoping to use 6.6 but there isn’t a DDK up for that yet.

Install DDK 6.5
Increase the memory (I used 4GB) or you’ll run into errors on the RPM rebuild with out of memory

on XenServer 6.5

install lsof
yum install –enablerepo=base lsof

you’ll want to use the DDK to develop an RPM as specified here –
Now you can download the HP stuff here – I wouldn’t I’d use the sources from FusionIO directly.  Download the Centos5 sources and all the Utilities from here –

Use WinSCP to upload the files to your DDK server
You should upload the Sources directory

run rpm –rebuild iomemory-vsl-*.src.rpm

Copy the rpm in the /usr/src/redhat/RPMS/x86_64 directory on the DDK to the XenServer
Also copy the utilities directory from the downloaded ZIP to the XenServer

rpm -Uvh iomemory-vsl-*.rpm (you may need to do this in order)

You can ignore the dracut error (search for it in the pdf here)

cd to Utilities – run the following (check the order I may be wrong here but it’ll tell you)
rpm -Uvh fio-common*.rpm
rpm -Uvh fio-preinstall*.rpm
rpm -Uvh fio-util*.rpm
rpm -Uvh fio-sysvinit*.rpm

now run


Note the driver version is empty? – now we are in sync with Kyrian’s blog post.

Run fio-status again – I had to upgrade the firmware on mine meaning I had to copy the firmware file *.fff to the XenServer and run


Then reboot and fio-status again. Now I have to wait for some low-level format but that’s ok…


I and listed what I had for /dev/fct* and performed the following command
fio-format /dev/fct0
fio-format /dev/fct1

Then attached them
fio-attach /dev/fct0
fio-attach /dev/fct1

After that fio-status looks good and we mount
xe sr-create name-label=”ioDrive00″ physical-size=320Gb type=lvm device-config:device=/dev/fioa
xe sr-create name-label=”ioDrive01″ physical-size=320Gb type=lvm device-config:device=/dev/fiob

And we are done!


Career, Citrix

CTP 2016!


I’m honored to be chosen as a CTP for the class of 2016!

It is exciting obviously for me, but also Champion Solutions to be able to speak on Citrix solutions for our customers! If you’re in Atlanta, feel free to reach out!

If you’re not sure what a CTP is, they are the top 50 Citrix experts or champions that are recognized for their contributions and technical expertise. There are only 50 people and this year included 6 new members. CTPs get to interact with the Citrix product team, provide input and also speak to the community. The other perk is they give you some discounts on Citrix Synergy which is in Las Vegas this year!

Thanks to all the current CTPs and congrats to all the new ones!

If you’d like to read more about the CTPs,  click here to read the current bios.

There is also a twitter list here.

Citrix, microsoft, vmware

I’ve got problems but 99.999 (five nines) storage isn’t one of them

I recently have been in front of a few customers discussing various designs for application and desktop virtualization.  Inevitably, or at some point, we discuss storage.  When it comes to storage I often pause and read the room because most people i know on the VAR and customer side have their favorites and have what I would refer to as a Dallas Cowboys team (I’m an Eagles fan, if you are a Dallas fan, just reverse the teams, it’ll work).

I’ve architected (is that a real word?) large deployments involving multiple datacenters, high availability and disaster recovery. My focus isn’t on what is the single best technology and gluing things together, it’s about what works (and hopefully, what works well).  Storage can be a very big issue with VDI, traditional SAN-based storage was not designed for desktop workloads and we’ve been oblivious to faster disk speeds and low latency on drives that hum under our wrists when typing.  Moving these workloads to the data center doesn’t always work and when you add in latency from a server reaching out to a separate SAN, it compounds the problem.

The traditional SAN isn’t usually the best fit for heavy desktops and applications, however, adding flash technology to the mix often deals with the IOPs issue and latency can be minimized.  Is flash necessary?  Nope.  I’ve had designs involving 15K SAS drives local to blades work very well.  The Citrix stream to memory, overflow to disk can perform even better with 10k or 7k drives.  However, I often don’t get to position that solution which brings me back to my first point…everyone has favorites.

I can take almost any storage and find a solution.  Even a traditional SAN, if I can use memory to cache, I can make that work.  Local disk? Easy.  Flash appliances, they are great!  But there is one thing I’m hearing that I don’t need.  The storage providing high availability or five nines.  There is a simple reason I don’t need five nines and I cringe when I hear others use it and lean back.

Your application doesn’t solely rely on storage to be available!

How will five nines prevent downtime when your hypervisor crashes or profile corruption occurs?  What about a failed backup on SQL that just eats up disk space?  What should we do?

We need to embrace failure and assume things fail.  It’s so much cheaper than having the hardware give you a warm fuzzing feeling.  When that business app fails, the business doesn’t care whether it’s storage or a cleaning person tripping over a server cord (I hope that isn’t even possible in most of your environments!). They see IT as the failure, not storage.

I wish I could take credit for this thought process but netflix has pretty much perfected this thought.  If you haven’t heard of the chaos monkey you should learn – .

Spend enough time in IT and you’ll realize that chaos always wins and you burn out quick if you’re fighting it.  However, returning to my original point, the design and architecture can do this also.  When we talk of desktops, many argue persistent versus non-persistent.  Persistent means you keep your desktop, non-persistent means you can roam (which usually means some flavor of roaming profiles).  I’m a big advocate of non-persistent.  Your storage or server fails, you get logged off, you log back in and you’re right where you were (or very close to it).  If the application is database driven and supports mirroring, you can survive storage failures, if setup correctly.

Going back to storage, this means two of whatever I have.  Two local drives, two appliances, two SANs.  I’ll take two 95% up-time appliances over a single 99.999% appliance anytime.  I’d rather save costs with single controllers than try to make a single point of failure not fail (because your application never has a single point of failure, it’s got multiple points of failure).

I’m not arguing five nines doesn’t have a place somewhere.  If you can’t use non-persistent, it might be for you.  However, I’d argue that virtualizing your applications and desktops is not a good move if you need persistence anyways.  Just my two cents, feel free to comment if you agree, disagree or think I’m full of it, I’m always open to suggestions!

PS – This is a first draft to publish, I’m sure there some typos and run-on sentences in there.


InfoBlox and Citrix Issues [RESOLVED]

I have heard a lot on Infoblox issues with Citrix and had the chance to meet some of the Infoblox team today for lunch and a meeting.  My first question, and Kevin Dralle’s, whom I work with, was about the apparent incompatibility of InfoBlox and Citrix, especially with PVS.  Please comment if you think this doesn’t work or has issues.


Some of the issue have been described elsewhere (I know Jarian Gibson has wrote and tweeted a few things on this also)

With Infoblox there is a CTX but it’s a bit mysterious on details

So what is going on with InfoBlox, anytime we have had customer with InfoBlox on and Citrix we cringe or opt for perhaps a dual NIC, isolated PVS VLAN (using Microsoft DHCP).  In any case here is what happens.

InfoBlox assigns the device a UID based on MAC but also on some of the device characteristics. So when we boot off PVS, we bring up the bin file which acts as the OS at the time of PXE boot.  We have a static MAC but since after the bin file pulls the TFTP image then brings up a windows OS, the UID changes, which infoblox assumes you’ll need another IP address.  Obviously there are use cases for this but for PVS this is an issue as you’ll get two IP addresses.

One fix has been to use reservations but this defeats the whole purpose of using an appliance or solution to manage this all.  Furthermore, when or if you get into automation and orchestration, you’ve got one more component to worry about when increasing the scope.

You do need to be on the 6.6 or higher release for this option but it is worth it if you have this issue or are an InfoBlox shop and want to rollout PVS without trying something uncommon to deployments (using BDM has a lot less collateral out there than PXE boot does).

Below are the two areas from the Grid and Member layers where you can set this (courtesy of InfoBlox!).