Linux, Red Hat, Uncategorized

Every time you disable SELinux God kills a kitten

Anything that states you must disable selinux to get it to work should never be run in production or on a system outside a lab.

This post is to enable you to run selinux when others don’t know how. This post will make you better with linux if you have had trouble with selinux.  This post may even improve your life and all sort of other stuff.

Why care about selinux?

Too many articles out there that default to “just disable selinux” and it works.  From my windows days, this is very much the same as the “just disable User Account Control “UAC” and the windows firewall.”  The obvious reason not to do this is security and although security is hard, it is important. SELinux provides additional security and mandatory access controls in linux.  It confines applications so they don’t compromise or harm other processes.  Another way to think about this, is that SELinux is written to secure apps that aren’t inherently secure or as secure as they could be.  In short, you want to be secure and you want your stuff to work.

This guide will show you how to uncover anything selinux is blocking and add policies to allow the behavior with some pretty cool tools available.

How do I make this all work?

It’s important to note selinux has 3 modes of operations.  One is disable, this turns off everything and really should never be used unless you are in a locked down environment and require the sliver of performance lost by adding selinux.  enforcing mode turns selinux on and enforces the security that is set.  permissive turns selinux on but does not enforce the rules.  Use permissive instead of disabled.  When anyone says disabled selinux, they should be saying, make selinux permissive because I don’t know how it works.

Security Hardening

This guide is designed to harden the security on centos, rhel, fedora and any system that can install the tools and use selinux. The changes suggested here may impact your system and cause services or features to become unavailable without proper testing. By following this guide and understanding what you are doing, you should be able to prevent this. This guide will show you how to apply SELinux to your system and be alerted for anything that is blocked.

Minimum requirements

  • Linux kernel supporting selinux (should be anything after 2.6) – ideally a VM or testing installation
    • RHEL, Centos or fedora (ideally RHEL 7 or Centos 7 or higher)
    • other distros can work but I’m using yum below (use dnf for fedora)
  • SSH or terminal access to your system
  • Familiarity with vim or your favorite editor
  • Ability to use 2 terminals at once or screen

Installation

We first want to set selinux to permissive. Before we do this, it’s best to have your disks created and the structure setup. Permissive allows us to log all SELinux violations without preventing their actions. This is good because we can see what will be blocked. In addition, we will install some tools that almost tell us exactly how to unblock anything that does get blocked.

Let’s go ahead and install the tools, set permissive and reboot for it to take effect (you can do this without reboot but you’ll miss any issues that present themselves on reboot).

Install the extra tools

# yum -y install setroubleshoot-server setools mcstrans

Now let’s enable selinux in permissive mode. We need to reboot after to catch the logs. You can use vi on /etc/selinux/config or copy/paste the line below (check those file paths are the same on your system)

# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config && cat /etc/selinux/config
# systemctl reboot

Now once the system reboots, you will get some log entries that we can find out if SELinux is blocking anything. We want to find entries that indicate something it being blocked. Let’s search the messages log.

# cat /var/log/messages | grep "SELinux is preventing"

If no errors, let’s search for just “SELinux”

# cat /var/log/messages | grep "SELinux"

We should get something back like this if everything is ok.

Aug 10 15:43:09 Rocky3 kernel: SELinux: Initializing.
Aug 10 15:43:13 Rocky3 kernel: SELinux: Disabled at runtime.
Aug 10 15:51:15 rocky3 kernel: SELinux: Initializing.
Aug 10 15:51:17 rocky3 kernel: SELinux: Disabled at runtime.
Aug 10 15:52:39 rocky3 kernel: SELinux: Initializing.
Aug 10 15:52:41 rocky3 kernel: SELinux: Permission validate_trans in class security not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_iscsi_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_fib_lookup_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_connector_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_netfilter_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_generic_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_scsitransport_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_rdma_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class netlink_crypto_socket not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Permission audit_read in class capability2 not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: Class binder not defined in policy.
Aug 10 15:52:41 rocky3 kernel: SELinux: the above unknown classes and permissions will be allowed
Aug 10 15:52:41 rocky3 systemd[1]: Successfully loaded SELinux policy in 69.596ms.

If something is being blocked we would see something like:

Aug 10 16:10:18 rocky3 setroubleshoot: SELinux is preventing /usr/sbin/nmbd from getattr access on the file /etc/samba/smb.conf

If you have any errors, please see the troubleshooting part of this document before enabling the enforcing mode. For clean installs without any additional services, you should not see any errors.

Once again, if you see errors, go to the troubleshooting section. If you installed a program, you’ll likely see selinux blocking something.

Let’s enable enforcing now that we have no errors.

# sed -i 's/^SELINUX=.*/SELINUX=enforcing/g' /etc/selinux/config && cat /etc/selinux/config
# setenforce 1

It is a good idea to now check for errors again but you should not see any if this was done right. We can run the sestatus command to check selinux status.  If you see errors, resolve them before you reboot!!!

# sestatus

Should show the following:

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 30

now let’s reboot and check once again to ensure selinux is enforcing and we don’t have additional errors

systemctl reboot
getenforce
sealert -a /var/log/audit/audit.log

Troubleshooting

Resolving log messages

Some other places to look for errors include utilizing the sealert tool, this tool not only shows the error but shows how to fix the error with confidence scores indicating how sure it thinks it is.

When we search in /var/log/messages, we may have a message similar to the one below. Note the last sentence, which says what command to run.

Aug 10 16:11:05 rocky3 setroubleshoot: SELinux is preventing /usr/sbin/nmbd from read access on the file smb.conf. For complete SELinux messages. run sealert -l 109d0656-c57b-4fdd-9ef3-09c6cee6351a

When you run that command, you’ll get an output similar to the one below. Note the confidence scores and resolutions. Look up what they do (don’t blindly trust everything please). The following output indicates that SELinux is preventing the smbd service from modifying the smb.conf file.

SELinux is preventing /usr/sbin/smbd from open access on the file /etc/samba/smb.conf.

                                            *****  Plugin restorecon (60.0 confidence) suggests   ************************

                                            If you want to fix the label. 
                                            /etc/samba/smb.conf default label should be samba_etc_t.
                                            Then you can run restorecon.
                                            Do
                                            # /sbin/restorecon -v /etc/samba/smb.conf

                                            *****  Plugin samba_share (30.8 confidence) suggests   ***********************

                                            If you want to allow smbd to have open access on the smb.conf file
                                            Then you need to change the label on '/etc/samba/smb.conf'
                                            Do
                                            # semanage fcontext -a -t samba_share_t '/etc/samba/smb.conf'
                                            # restorecon  -v '/etc/samba/smb.conf'

                                            *****  Plugin catchall_boolean (5.26 confidence) suggests   ******************

                                            If you want to allow samba to export all ro
                                            Then you must tell SELinux about this by enabling the 'samba_export_all_ro' boolean.

                                            Do
                                            setsebool -P samba_export_all_ro 1

                                            *****  Plugin catchall_boolean (5.26 confidence) suggests   ******************

                                            If you want to allow samba to export all rw
                                            Then you must tell SELinux about this by enabling the 'samba_export_all_rw' boolean.

                                            Do
                                            setsebool -P samba_export_all_rw 1

                                            *****  Plugin catchall (1.09 confidence) suggests   **************************

                                            If you believe that smbd should be allowed open access on the smb.conf file by default.
                                            Then you should report this as a bug.
                                            You can generate a local policy module to allow this access.
                                            Do
                                            allow this access for now by executing:
                                            # grep smbd /var/log/audit/audit.log | audit2allow -M mypol
                                            # semodule -i mypol.pp

To resolve this, we want to have smbd access to the smb.conf file. We are saying that service can now access it. SELinux was protecting us from some service just using that file or modifying it, which is a good thing! In the samba section we will run the commands to enable smb.conf access and also read/write if desired.

sealert usually targets a log file with the -a option. These usually will give you the messages you need.

# sealert -a /var/log/audit/audit.log

Samba

When you enable Samba, you’ll start getting audit messages.  This is because it doesn’t allow modifying smb.conf which is what you are doing (you’re adding lines to the bottom of the file indicating what shares you want to export). To allow this we need to allow access and also allow read only or read/write.

# semanage fcontext -a -t samba_etc_t /etc/samba/smb.conf
# restorecon -Rv /etc/samba/
restorecon reset /etc/samba/smb.conf context system_u:object_r:tmp_t:s0->system_u:object_r:samba_etc_t:s0

We just set the security context on smb.conf so it is able to be used. Restorecon is necessary because it sets the contexts correctly on the files and directories.

Below is a blanket boolean to allow samba rw/ro access. It would be better to set this for the directories you share rather than all over but for now, we will just set it. Use ro for read only or rw for read/write

 # setsebool -P samba_export_all_ro 1

OR

 # setsebool -P samba_export_all_rw 1

NFS

I haven’t found anything blocking NFS yet.

Docker

These will vary by container so anytime you add a container you should check sealert or grep in /var/log/messages for an error.  In general, this is more likely to come up with persistent volume mapping or exposing external ports. The other area to keep an eye on is sharing volumes between containers. This worked for me but I would always check after enabling.

Importing new pools or adding disks or btrfs

This is going to be the scariest area here. SELinux adds contexts to files and directories and if this gets messed up, you won’t be able to access much. BEFORE you do this, I would likely just go into /etc/selinux/config and set everything to permissive just in case. Do the import, double check the logs then set it back to enforcing.

Career, Uncategorized

What would you say you do here?

 

When people ask me what I do it always puts me on defense trying to figure out how much knowledge you could handle and then speaking within a few seconds.  I heard a colleague, John Willis, speak and he discussed the standard response of “computers” which is a solid answer to avoid a long discussion.  Even better, a former colleague, who was an amazing saleswoman at EMC, used to say “I sell bananas!” when it was clear there was nothing happening.

I’ve thought about this a lot and kept thinking back to Officespace with the Bobs asking Tom Smykowski the question above.  Tom responds with a painfully obvious “nothing useful” that he seems to realize as he says it.  Basically he brings the specs to the engineers from the customers so the engineers don’t have to. When the Bobs ask why can’t the engineers bring them to the customers themselves, Tom shouts that he is a people person, gets flustered and exhorts, “What is wrong with you people!”.

While I don’t think my job or most others falls into this, it’s important to be able to describe what you do.  There is a saying that if you truly understand a subject, you can explain it to anyone, even a child.  I believe this to be true and I think it’s important that those in IT be able to understand their job and be able to explain it.

If you’re wondering, I have tried explaining what I do to my kids and wife with varying degrees of success.  I’d also encourage you to do the same.  If you get flustered or worried, I think you have every right to circle back to your manager or senior level company person and get some answers.  If they don’t know, take advantage of that open-door policy the CIO or CEO always brags about.

In DevOps, I always stress Value Stream Mapping.  This is a process where we take a workflow that involves our business (like a customer ordering a product and us making it) and we break it up into boxes, visually, and look at cycle times, delays and where we are in the process.  It’s important to do this visually because we don’t always see the big picture clearly when we are on the floor or entrenched in work.  It’s also invaluable because it shows where you add value (or where you don’t) and gives you the opportunity to change that.  You can see that when I do this, I do not add any value to the overall goal, or you may find that you are adding tremendous value and not realizing it.  In any case, the goal is to understand how your company works, how it makes money and where you are in the process.  If you take this and apply it to IT, you should find that your role is a part of the business and does add value.  If not, or if no one knows how your company works, you should start asking questions and looking for better opportunities.

Today I work at Red Hat. Red Hat takes community projects and turns them into enterprise products.  If that’s hard to grasp we take free software and add support and stability for companies to use and be able to trust.  I get paid based on the sales made from the 2 salesmen I support.  Our products help businesses focus more on their business and less on IT.

Next time you want to say computers, try to explain it.  If you fail, you’ll get another shot.  However I think in time, you’ll either learn to explain what you do to anyone or realize you can do something a lot more worthwhile.

 

Citrix, Uncategorized

Citrix Synergy as CTP

Getting to attend Synergy 2016 is one of the perks that accompanies being a CTP.  You also have responsibilities that come with this but I’m finding the perks seem to outweigh them!

I joined Red Hat last week and I think I’m the only Red Hatter here (I get weird looks with my hat but that’s on purpose).  Citrix is heavily aligned with Microsoft, Microsoft announced Red Hat on Azure and there has always been interest in VDI with Linux.  That being said, I get to come to Synergy as more an outsider than ever before (instead of from the customer or partner side).

CTPs have quite a few meetings lined up when they come to Synergy that start 2 days earlier than the kickoff.  These meetings are no joke and I got plenty of warnings about pacing myself.  They started at 8am promptly and continued to about 7pm two days in a row with few breaks.  The topics are intense and interesting and it’s easy to get overwhelmed.

The meetings are generally with the Citrix product teams and we get to learn and give feedback to new releases, ideas, successes and failures.  Each team is different in terms of how they operate.  I can’t go into too much detail but Citrix is very interested in improving their feedback loops and ensuring that products developed have purpose instead of just throwing things over a wall and surprising people.  This is a good move as it allows a lot more trail and error, and a lot less of “what the f is this thing?” when something in announced.  This is definitely the primary interest I had with my CTP meetings, in addition to the awesome company!

This year I’m not speaking at a breakout but doing some other activities.  I’ll be at the Synergy Showdown #XAonAzure with my team, the Village Idiots, along with Esther, Jarian and Paul from the CTP team.  We will be taking on the wily dutch team but our secret weapon hails from the Netherlands (Esther) so we are a lock in to win 😉 !  I will also have a few match.geek sessions if you want to hear me blather on about whatever you want and I’m hosting the education tech chat tables (except for today, Tobias will be doing Tuesday’s due to the showdown).

Today was fun, I got to sit at the bloggers table and get super secret early access the the keynote.  Esther even got me a front row table seat and the view is spectacular.  If you don’t think a CTP has perks, you’re mistaken and this is one of the best perks I’ve ever had.

If you’re here and want to connect, let me know. You can look for a dude wearing a red hat (literally) and it’s likely me or DM on twitter or the Synergy app. Looking forward to the rest of the week and running into old friend and making new ones!

Career, Red Hat, Uncategorized

First week at Red Hat

I recently joined Red Hat on May 16th and spent a few days in Raleigh at orientation.  I can’t think of a time before that I felt as good a fit as at Red Hat.  Don’t get me wrong, this isn’t a job to sit on your ass, however, if you work hard, you are rewarded.  The company culture seeps out of people and as best I can describe it, it is basically one of common sense.  Nothing overly crazy, but a lot of thought put into decisions, then once made, they are implemented quickly.

For any of you wondering why I made this move, I had spent the last year at Ericsson working on a very large OpenStack rollout with AT&T.  What we did was great but, my heart lies with presales and I couldn’t pass this opportunity up.  I’ve always tinkered and been a hobbyist with Linux and love the thought of having a career with it.

I have no idea how much I can weld my past technologies with what I am doing today but time will tell and you have to make changes in life.  Nevertheless, I am at Citrix Synergy next week in Vegas, so if you’re out there let me know and we can meet.

OpenStack, Uncategorized

OpenStack Summit – keynotes

Thanks to Champions Solutions Group, I’ve been able to attend the OpenStack Summit this year, 2016, in Austin, TX.  The event is interesting considering my experience with other conferences.  Generally I attend those that are geared towards partners, customers and sales. However, OpenStack is geared to a wide audience, but especially towards the engineers, architects and users which makes me feel right at home.

Highlights included the attendence of 7500 people and also the revelation that 65% of deployments of OpenStack are now production deployments!

OpenStack’s keynotes focused on a few themes.  Donna Scott from Gartner was correct in pointing out “disruption” as a key motivator explaining bimodal methods of deployment (mode 1 being reliable stable and mode 2 more experimental but agile). AT&T presented their challenges and wowed the audience with the rate of growth of mobile data (150,000% from 2007-2015!).  They are adopting a white box solution to lower cost, increase speed and agility. They also talked about a management layer for mult-tenant, complex networks.

Mirantis had a great presentation and poked at Gartner on their past of saying OpenStack wasn’t a real player (but that it’s in the past and happy they are hear).  After some story about Russian and vodka drinking bears, Mirantis made some excellent points.  Namely, people and process over tools and technology.  This is exactly the core of DevOps and if you know me, is a good overall view (I wish they went into value stream mapping a bit but it’s a start!).  They also pointed out that while they are a big AWS fan, there is ENORMOUS market share outside AWS and AWS has a teeny part of it!  They also poked a little fun at the vmware guys (not all, I know many are embracing change) but that adopting agile and openstack without looking at people is a killer.  Namely public cloud adoption seems great because you don’t have to deal with the people, but it’s not cheaper or always better.

We also heard from SAP and VW and Red Hat.  Red Hat stayed on message with the 65% of deployments to production.  SAP explained their addressing the movement of the industry with a customer expecting instant response time with a tweet. VW explained the connected car and the use of cloud computing.

Day 2 had keynotes that focused on containerization.  QUite a lineup of presenters including Cisco, OVH, TWC and liveperson and smartcities (and more) spoke. I think the best highlight was the presentation by Google and CoreOS highlighting tectonic which overlays Kubernetes (https://tectonic.com/).  I may have got this wrong, but each kubernetes node can hold many containers and they run openstack components as containers. They demonstrated a horizonv2 to v3 upgrade and also self-healing by killing a neutron and then a nova node.  I was definitely impressed and need to shore up by kubernetes skills!

 

Citrix, PVS, Uncategorized

Citrix PVS and Managed Service Accounts gMSA

I’m a big fan of Managed Service Accounts because they are much more secure and aren’t easily exploited by human beings.  Basically, Active Directory controls the account with it being responsible for changing passwords.  While use of gMSA (group managed service accounts) is sometimes hit or miss, I didn’t find much on recent use with Citrix other than a vague “we support this” statement.

Carl Webster had a much older attempt with PVS (not sure whether he tried again or not) and I wanted to ensure that this worked on PVS 7.7 (just released).

You’ll need a couple of things

I would leverage a tool for creating and managing gMSA that I got here.
(note: for a quick guide on setting this up, I would look through Derek Seaman’s blog).

PVSgMSA

Add you PVS server to the list otherwise it won’t work. (I only have 1 PVS server right now, I’m in rebuilding mode…)

PVSgMSAComputer

For SOAP, you’ll need to make this account a member of the local admins on the PVS server (when you add the account, make sure you select “service accounts” for objects.

LocalAdmin

For SQL, I am using 2014 with availability groups.  Check out Derek’s blog for a great walkthrough on this.

Your database should have been created already (use the dbscript.exe to manually create the database in PVS)

Grant the permissions needed to your gMSA on the SQL database (I create the account on both database servers just in case (when I test the failover))

Testing failover should work and you will also notice the services are runningpvsconsoleservicespvs