19 October Saint John of Rila Тhe Wonderworker – A little known Hermit saint of Great importance and fervent prayer intercessor for humanity


October 20th, 2021

Saint-Ioann_John_of_Rila_Rylsky_icon

I've already blogged a number of times articles of Saint John of Rila as he is The Greatest venerated Bulgarian Saint as of today his birthplace nearby today's Ruen Monastery as well as my inspiration from the Night Vigil prayer gatherings in the Monastery in his Memoriam in German. But I'm pretty sure this spiritual giant  is under estimated and little known to the public not only the non-bulgarian English speakers but even to Bulgarian and the saint deserves much greater veneration and honour than it has received through the ages for the fact he has been among the greatest prayer intercessors for the world until the beginning of ages.

For his immerse hermit achievements in a similar way as Saint Anthony the Great he has been granted All the innumerous Gifts of The Holy Spirit.
The humble hermit even in his life time he has been defined as "Angel in flesh" and a "Citizen of heaven".
The life of Saint John has been a X-th century continuation of the life of Saint Simeon the Stylites and Saint Simeon Stylites the Younger attempted to be repeated as far as possible by more contemporary saints such as Saint Siluan the Athonite.
 

He has been a teacher (even though he ran away from people) all his life. He has been the major establisher of Monasticism in Bulgaria, 

In 18 of August year 946 Anno Dommini saint John has departed this world. Dying with a painless way called in Church language with the term Dormition or Assumption – (meeling he did not felt any kind of fleshly pain as normally one goes through when he ends his life unsanctified). His Dormition  in a similar way as The Mother of God (Theotokos), an end of life who has been granted throughout human history only to few other saints.

Assumption_Dormition_of_Saint_John_of_Rila_by_Toma_Vishanov

Saint John who spend his hermit lifetime in Mountain of Rila was born circa y. 876 AD and passed away to Christ on August 18, 946 A.D.
Thirthy four years after his dormition, he appeared in a dream to his pupils and has commanded them to dig out his incorruptable body and move it to in Sredetz (today the city of Sofia capital of Bulgaria). In Sredetz the holy relics of the saint has been kept in honour for 200 years.

Saint John has been a contemporary to Saint King Boris I the Baptizer of Bulgaria, King Simeon the Great (Son of King Boris, who raised Bulgaria to Empire reaching to 3 Oceans, who has been one of the largest empires of All times) and King Petar (Peter) the First. His glory as a Wonderworker who has been desired to been seen by many to receive physical healing or spiritual advice has been mostly through the rule of King Petar I (927 – 969). In Medieval times the Glory of Saint John of Rila has been widely known throughout all the Christianized Lands (including Byzantine Empire) and the West in the Catholic Church. The implication of his importance as a medieval healer saint and a model for hermit life and saintship and image for truthful spiritual life is well seen as his Biography (Living) is seen as he has about 15 Known different livings saying varyious details about his lifetime written in different centuries in Old Bulgarian Language and in Middle centuries in the new transformed form of Old Bulgarian writting called Church Slavonic, where saint has been simply adressed by monk name Ioan (John).

According to his biography we found until the age of 25 Saint John has been a simple shepherd. Until the yough age in his heart John had the burning love in his heart and a desire to dedicate himself completely to God. When his parents has passed away, he gave away all his received inheritance to the poor and sick people and entered as a aprentice in the monastery of today Boboshevski Saint Dimitar under the Ruen Monastery in the Mountain of Vlahina.

There in he received a Theological education, studied the liturgical books, received a spiritual dignity and prepared himself for the great spiritual mission of hermitage and life long prayer for the world. Receiving the monks schima, he gave away completely to fasting and prayer, initially establishing himself in Vitosha Mountain (most likely in today Monastery of German Monastery St. John of Rila (convent of Mount Athos Monastery of Zograph) nearby Sofia

Saint_Ivan_Rilski_Zemen-ikona

St. John of Rila Wall Painting XIV century from the Church Zemen Monastery Saint John the Theologian

In German Monastery often times the ancient Christian tradition of Night Vigil is followed on hist feast Major Christian feasts even today.
Later he has been for a while in the small Village  "Dobarsko" near The Europe Famous Resort of Bansko.
A short after 20 km of Dobarsko he has established The Most Famous Monastery for the Balkans, the Rila Monastery, a pivotal place whose light has shined upon practically all parts of the land as it become a school for tens of thousands hundred monks throughout the last 20 centuries. Many manuscript works prepared in Rila Monastery has been later on transferred in Today Ukraine, Russia, Serbia, Greece and even the Western Catholic lands.
There is no exact place in history when Saint John was officially canonized as in Medival Times, saints did not go through an official canonization but has been testified by the multitudes of peoples who has been cured by their wonder-working prayers. By the prayers of Saint John many has been able to conceive childs, leppers get cured and could walk, eye diseases were healed, full health has been restored or the gift of faith has been giving, not to mention the multitudes of people who came to Christian faith thanks to the prayers of St. John.

The-Meeting-of-Saint-John-of-Rila-and-King-Peter-First-of-Bulgaria

King Petar I-st meets the Hermit Holy Father John of Rila (Wall Painting)

The story of the powerful medieval Bulgarian ruler Peter the First who travelled 450 Kilometers only to see the saint is well known. Travelling about 450 kilometers from the capital of Bulgaria Veliki Preslav to the Cave of Saint John of Rila only to meet him.

Cave-of-Saint-John-of-Rila-Peshterata-na-sv-Ioan-Rilski

The Cave of Saint John of Rila Cave exit

The monk however refused to meet him due to his immerse humility, where King Peter I-st was able to talk in a long distance via a valley with the saint, offering him gold and fruits. St. John refused to receive the king's gold, as he said he did not left the world and everyhing to it to attain gold or anything but to save his soul. He received the fruits not wanting to insult the king. Many people who deserved to learn a true monasticism and ascetism has quickly reached the saint and asked them to create a monastic brotherhood and wanting to be useful his poor brothers and having revelation of God that it is the will of God to establish the monsatery he accepted and become the first Abbot of Rila Monastery.

He passed to the Savior Christ about in year on 18 of August 946 AD being aged about 70 years old he has been buried in the Church porch in a stone tomb, preserved even today.

The-Testament-of-Saint-John-of-Rila-Given-to-his-pupils-monks
Saint John Giving his Written Testament to his Monks (The Testament is Famous for the quote "I beg you children to mostly fear from the snake of avarice which is the root of all evils".


Until his death he has departed from the brotherhood in complete solitude and wrote his famous "Testament".  The Testament of Saint John of Rila he left for his pupils on 25 of March on the feast of the Annunciation in year 941. The text of the Saint John Testament is preserved in a copy manuscripts dated from the XVIII and XIX century and contains a multitude of quotes from The Scriptures of Old and New Testament

Engravement-of-the-Living-of-Saint-John-of_Rila_monastery-icon

The Living of Saint John of Rila


Saint_John_of-Kronstadt-Ioann_Kronshtadtskij

 

A little known fact is that the Russian All Famous Saint Father John of Kronstadt (also a wonderworker) (Ivan Ilyich Sergiyev, born: 31 October [O.S. 19 October] 1829 + Passed to Christ: 2 January 1909 [O.S. 20 December 1908]) has received his Baptizmal name after Saint John of Rila as he has been born on his feast day and his parents following the Russian tradition to name the baby after the Saint honour in the Church Calendar gave the name John of St. John of Kronstadt. St. John of Kronstadt deeply venerated Saint John of Rila and asked his prayer intercession alway until his life time.
​In 1900, Father John established the women's monastery of Ioannovsky Convent, named for his name patron John of Rila, as a branch of the Sura Monastery of St. John the Theologian, where he was later also buried.

Recognition of Saint John of Rila Holiness shortly after his death


After his death in y. 946, st John has been buried nearby the established by him Holy Rila Convent, shortly after King Peter I-st ordered to be transferred to Sredetz (Sofia). Most likely this can be considered for original date of canonization of the saint. About y. 989 – 992 y. his holy relics has been transferred by King Samul and then Patriarch German-Gabvril (German-Gabriel) in Sredetz. This happened on 19 of October and since then on this date is venerated his bright memory.
Initially his holy relics has been kept for venerateion in the Church of "Saint Great Martyr Georgi the Glorybringer" and later on moved to Church of "Saint Apostle and Evangelist Luke". In the XII century in his honour the Church of "Saint Luka (Saint Luke)" was built near which a monastery has been established. The Byzantine writter Ioannis Skylitzes / John Skylitzes testifies that in Sredetz the Holy Relics of Saint John has miraculously cured emperor Manuel I Komnenos.

In 1183 during the consequential Hungarian-Byzantine war, the Hungarian King Bela III of Hungar conquered Sredetz and moved the holy relics of Saint John in his capitzal Estergom today the Komárom-Esztergom County. According to tradition the local Roman Catholic archibishop declared he has not been aware for the existence of a saint John of Rila and because of his words the saint has punished him with immediate numbness. Once he venerated the holy relics of the reliquary and begged for forgiveness, his speech was recovered. Amazed and worried about this miracle, in year 1187 hungarians transferred back the holy relics of the saint to Tarnovo? Bulgaria.

The-Return-of-Holy-Relics-of-Saint-John-of-Rila

The Return of the Holy Relics of St. John to Trnovo Bulgaria

On 1st of June the Church celebrates the return of the holy relics of saint John of Rila from Tarnovo to Rila monastery. In 1195 Bulgarian king Ivan Asen the First solemnly transferred the Holy relics once again to Tarnovo in the church still being built St. 40 Martyrs (dedicated to 40 Martyrs of Nikomidia). Until the church was completed the holy relics has been kept on Krastec Peak, now the place has a small chapel in honour of St. John of Rila The Wonderworker".

In 1469 year Bulgaria is already invaded by the Ottoman Turks, thus  after a permission from the Sultan, the monks of Rila convent brought the holy relics of the saint from Veliko Tarnovo to his established Rila Monastery, where holy relics are kept to this date and where anyone who needs a spiritual fortification, restorement of health, or suffering from unclean spirits (devils) could ask with faith and hope for healing for the saint and wait for the miracle.
I myself have had the chance to meet a man who has being to a Night Vigil to Rila monastery and testified that he and his life both being in his old age of fifties after not being able to have kids and trying out everything has asked the saint during a night vigil to pray for them to able to conceive a child and the saint heard their prayers and granted them a kid and she conceived right after that and brought to the earth after 9 months a very healthy and lively baby 🙂

Another modern miracle I'm personally aware of made by Saint John is the Gift of Faith in God which was granted to a Greek architect, I've personally met while being on a pilgrimage visit to Holy Mount Athos. The Greek shared that he had never been a believed in God and was a grown atheist, until he had a work duty to do some reconstruction restoration works in Bulgarian Rila monastery. Spending a few months as part of his business-trip duties, he has received the gift of faith. I've met the Greek (unfortunately no longer remember his name), nearby the monastery of Xenophontos.

 

The troparion of our rev. father John of Rila, tone 1

A foundation of the repentance, an example of the tenderness,
a model of the consolation and of the spiritual perfection,
oh Reverend Father, was your equal to the Angels life therefore
staying in prayers and fasts and in tears
pray to the Christ God for our souls.


Today the holy relics are miraculously still incorruptable and emit a specific nice odor  which is typical for many of other saints, a testify to the unbelievers for their saintship. The saints relics in a special reliquarium in the right corner behind the iconostasis icons. The right hand of the saint is kept in nun convent in Island of Tinos Greece and the story how it ended there is a worthy for a small pamphlet and I'll try to say few words on that in some of the next posts.

Let by the Holy Prayers of Saint John of Rila our Home Land Bulgaria and all Countries are granted more Peace, Love, Faith and Kindness and relief from the terrible diseases that are plaguing the world today !

Holy Father John of Rila Pray the mercyful Lord Jesus with All the Saints and The Most Holy Theotokos to save our Souls and grant mercies to us the sinners !

Saint protector of the Family and The receipt of Saint Petka Tarnovska by King Asen II story


October 14th, 2021

Chapel-Trun-Sveta-Petka-Ikona-ot-16-17-vek-na-Sveta-Petka-v-skalniq-paraklis-na-Trun-231x300

In these hazy and confused times where the family as institution is failing and it is becoming more and more modern for people to live together without official Civil marriages count are steadily declining not to mention that the Church marriages no matter whether it is a Protestant, Roman Catholic is very rare thing. The attack against families is multi-vector one, it is descredited and being pranked on the TV in movies and serials, in the press. Many of the bad sides of marriage are exposed as well as the incapabilities of any traditional marriage to respond to the modern challenges of the world and hence many choose to not marry. But originally God created man to live in a family Adam and Eve were the first marital couple (even though they did not officially binded it on paper) in the municipality or the Church. Of course there situation was slightly different than today as they were the only couple in the beginning when God created man one would say. But even after that through the Ages Marriage and fidelity that stems out of it has been considered normal. However today normality is being pranked and abnormality is being enforced from all sides. 
As the topic of family does directly concern me as I have a family I thought therefore it is useful to mention again today about the feast of Saint Petka Epivatska (Epivates) famous as Saint Petka of Tarnovo who is considered in our Eastern Orthodox Christian countries as a protectress of family.

The reason why we venerate saint Petka here in Bulgaria is because here holy relics has stayed in Tarnovo for almost 2 Centuries and being in Bulgaria, they have worked many miracles, many of which were done over a family husbands and wifes who had their usual trials, like infertility, problems in family, quarrels etc.
For being famous for that miracles she has been invoked for centuries worldwide by believing Christians for help on their family trials.

Below is shortly the history of how St. Petka Epivatska, holy relics were transferred to Tarnovo and the multiple transfers of her relics until she finally choose to reside in Iash Romania.
 

+++

Posreshhane_na_moshhite_na_sv-petka_v_Trnovo-Bylgariq
The reception of Holy Relics of Saint Petka Paraskeva in medieval Capital of Bulgaia Tarnovo

In 1230 King Ivan Asen II the most powerful South-eastern European ruler demanded from the the Knights of the Crusaders to submit him her holy relics who are found still in Tracian city Kaliakratea ruled at that time by the Holy Latin Empire. King Ivan Asen II together with the patriach Joachim the first receives her holy relics with honor and settles her incorruptabilities into the newly creates Church in honour of herself St. Petka behind Tsarevets FortressSaint Petka became from that point considered as a protectress of the city, the throne and the country.
Her holy relics arrived from Kallikrateia in Tarnovo, the Capital of Second Bulgarian Empire in year 1230 AD, she has been thus called Paraskeva of Tarnovo and has been venerated as a protectress of the Tarnovo city the Bulgarian nation and the country. The attitude towards Saint Petka Tarnovska as a protectress of Bulgarian nation and contry is been clearly seen by the mention in the Bulgarian and International acts (documents) and manuscripts of that XII – XII century.

To learn more about Saint Petka  Parskeva Epivates of Thrakia feast day today 14 of October check my previous article here.

Let by the Holy Prayers of Saint Petka the Families be granted grace to endure the hardships of life! Saint Petka pray Christ for us!

Apache disable requests to not log to access.log Logfile through SetEnvIf and dontlog httpd variables


October 11th, 2021

apache-disable-certain-strings-from-logging-to-access-log-logo

Logging to Apache access.log is mostly useful as this is a great way to keep log on who visited your website and generate periodic statistics with tools such as Webalizer or Astats to keep track on your visitors and generate various statistics as well as see the number of new visitors as well most visited web pages (the pages which mostly are attracting your web visitors), once the log analysis tool generates its statistics, it can help you understand better which Web spiders visit your website the most (as spiders has a predefined) IP addresses, which can give you insight on various web spider site indexation statistics on Google, Yahoo, Bing etc. . Sometimes however either due to bugs in web spiders algorithms or inconsistencies in your website structure, some of the web pages gets double visited records inside the logs, this could happen for example if your website uses to include iframes.

Having web pages accessed once but logged to be accessed twice hence is erroneous and unwanted, and though that usually have to be fixed by the website programmers, if such approach is not easily doable in the moment and the website is running on critical production system, the double logging of request can be omitted thanks to a small Apache log hack with SetEnvIf Apache config directive. Even if there is no double logging inside Apache log happening it could be that some cron job or automated monitoring scripts or tool such as monit is making periodic requests to Apache and this is garbling your Log Statistics results.

In this short article hence I'll explain how to do remove certain strings to not get logged inside /var/log/httpd/access.log.

1. Check SetEnvIf is Loaded on the Webserver
 

On CentOS / RHEL Linux:

# /sbin/apachectl -M |grep -i setenvif
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
 setenvif_module (shared)


On Debian / Ubuntu Linux:

/usr/sbin/apache2ctl -M |grep -i setenvif
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-default.conf:1
 setenvif_module (shared)


2. Using SetEnvIf to omit certain string to get logged inside apache access.log


SetEnvIf could be used either in some certain domain VirtualHost configuration (if website is configured so), or it can be set as a global Apache rule from the /etc/httpd/conf/httpd.conf 

To use SetEnvIf  you have to place it inside a <Directory …></Directory> configuration block, if it has to be enabled only for a Certain Apache configured directory, otherwise you have to place it in the global apache config section.

To be able to use SetEnvIf, only in a certain directories and subdirectories via .htaccess, you will have defined in <Directory>

AllowOverride FileInfo


The general syntax to omit a certain Apache repeating string from keep logging with SetEnvIf is as follows:
 

SetEnvIf Request_URI "^/WebSiteStructureDirectory/ACCESS_LOG_STRING_TO_REMOVE$" dontlog


General syntax for SetEnvIf is as follows:

SetEnvIf attribute regex env-variable

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] …

Below is the overall possible attributes to pass as described in mod_setenvif official documentation.
 

  • Host
  • User-Agent
  • Referer
  • Accept-Language
  • Remote_Host: the hostname (if available) of the client making the request.
  • Remote_Addr: the IP address of the client making the request.
  • Server_Addr: the IP address of the server on which the request was received (only with versions later than 2.0.43).
  • Request_Method: the name of the method being used (GET, POST, etc.).
  • Request_Protocol: the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.).
  • Request_URI: the resource requested on the HTTP request line – generally the portion of the URL following the scheme and host portion without the query string.

Next locate inside the configuration the line:

CustomLog /var/log/apache2/access.log combined


To enable filtering of included strings, you'll have to append env=!dontlog to the end of line.

 

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

You might be using something as cronolog for log rotation to prevent your WebServer logs to become too big in size and hard to manage, you can append env=!dontlog to it in same way.

If you haven't used cronolog is it is perhaps best to show you the package description.

server:~# apt-cache show cronolog|grep -i description -A10 -B5
Version: 1.6.2+rpk-2
Installed-Size: 63
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Depends: perl:any, libc6 (>= 2.4)
Description-en: Logfile rotator for web servers
 A simple program that reads log messages from its input and writes
 them to a set of output files, the names of which are constructed
 using template and the current date and time.  The template uses the
 same format specifiers as the Unix date command (which are the same
 as the standard C strftime library function).
 .
 It intended to be used in conjunction with a Web server, such as
 Apache, to split the access log into daily or monthly logs:
 .
   TransferLog "|/usr/bin/cronolog /var/log/apache/%Y/access.%Y.%m.%d.log"
 .
 A cronosplit script is also included, to convert existing
 traditionally-rotated logs into this rotation format.

Description-md5: 4d5734e5e38bc768dcbffccd2547922f
Homepage: http://www.cronolog.org/
Tag: admin::logging, devel::lang:perl, devel::library, implemented-in::c,
 implemented-in::perl, interface::commandline, role::devel-lib,
 role::program, scope::utility, suite::apache, use::organizing,
 works-with::logfile
Section: web
Priority: optional
Filename: pool/main/c/cronolog/cronolog_1.6.2+rpk-2_amd64.deb
Size: 27912
MD5sum: 215a86766cc8d4434cd52432fd4f8fe7

If you're using cronolog to daily rotate the access.log and you need to filter out the strings out of the logs, you might use something like in httpd.conf:

 

CustomLog "|/usr/bin/cronolog –symlink=/var/log/httpd/access.log /var/log/httpd/access.log_%Y_%m_%d" combined env=!dontlog


 

3. Disable Apache logging access.log from certain USERAGENT browser
 

You can do much more with SetEnvIf for example you might want to omit logging requests from a UserAgent (browser) to end up in /dev/null (nowhere), e.g. prevent any Website requests originating from Internet Explorer (MSIE) to not be logged.

SetEnvIf User_Agent "(MSIE)" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


4. Disable Apache logging from requests coming from certain FQDN (Fully Qualified Domain Name) localhost 127.0.0.1 or concrete IP / IPv6 address

SetEnvIf Remote_Host "dns.server.com$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


Of course for this to work, your website should have a functioning DNS servers and Apache should be configured to be able to resolve remote IPs to back resolve to their respective DNS defined Hostnames.

SetEnvIf recognized also perl PCRE Regular Expressions, if you want to filter out of Apache access log requests incoming from multiple subdomains starting with a certain domain hostname.

 

SetEnvIf Remote_Host "^example" dontlog

– To not log anything coming from localhost.localdomain address ( 127.0.0.1 ) as well as from some concrete IP address :

SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog

SetEnvIf Remote_Addr "192\.168\.1\.180" dontlog

– To disable IPv6 requests that be coming at the log even though you don't happen to use IPv6 at all

SetEnvIf Request_Addr "::1" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


– Note here it is obligatory to escape the dots '.'


5. Disable robots.txt Web Crawlers requests from being logged in access.log

SetEnvIf Request_URI "^/robots\.txt$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

Using SetEnvIfNoCase to read incoming useragent / Host / file requests case insensitve

The SetEnvIfNoCase is to be used if you want to threat incoming originators strings as case insensitive, this is useful to omit extraordinary regular expression SetEnvIf rules for lower upper case symbols.

SetEnvIFNoCase User-Agent "Slurp/cat" dontlog
SetEnvIFNoCase User-Agent "Ask Jeeves/Teoma" dontlog
SetEnvIFNoCase User-Agent "Googlebot" dontlog
SetEnvIFNoCase User-Agent "bingbot" dontlog
SetEnvIFNoCase Remote_Host "fastsearch.net$" dontlog

Omit from access.log logging some standard web files .css , .js .ico, .gif , .png and Referrals from own domain

Sometimes your own site scripts do refer to stuff on your own domain that just generates junks in the access.log to keep it off.

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)|(eot)$" dontlog

 

SetEnvIfNoCase Referer "www\.myowndomain\.com" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

6. Disable Apache requests in access.log and error.log completely


Sometimes at rare cases the produced Apache logs and error log is really big and you already have the requests logged in another F5 Load Balancer or Haproxy in front of Apache WebServer or alternatively the logging is not interesting at all as the Web Application served written in ( Perl / Python / Ruby ) does handle the logging itself. 
I've earlier described how this is done in a good amount of details in previous article Disable Apache access.log and error.log logging on Debian Linux and FreeBSD

To disable it you will have to comment out CustomLog or set it to together with ErrorLog to /dev/null in apache2.conf / httpd.conf (depending on the distro)
 

CustomLog /dev/null
ErrorLog /dev/null


7. Restart Apache WebServer to load settings
 

An important to mention is in case you have Webserver with multiple complex configurations and there is a specific log patterns to omit from logs it might be a very good idea to:

a. Create /etc/httpd/conf/dontlog.conf / etc/apache2/dontlog.conf
add inside all your custom dontlog configurations
b. Include dontlog.conf from /etc/httpd/conf/httpd.conf / /etc/apache2/apache2.conf

Finally to make the changes take affect, of course you will need to restart Apache webserver depending on the distro and if it is with systemd or System V:

For systemd RPM based distro:

systemctl restart httpd

or for Deb based Debian etc.

systemctl apache2 restart

On old System V scripts systems:

On RedHat / CentOS etc. restart Apache with:
 

/etc/init.d/httpd restart


On Deb based SystemV:
 

/etc/init.d/apache2 restart


What we learned ?
 

We have learned about SetEnvIf how it can be used to prevent certain requests strings getting logged into access.log through dontlog, how to completely stop certain browser based on a useragent from logging to the access.log as well as how to omit from logging certain requests incoming from certain IP addresses / IPv6 or FQDNs and how to stop robots.txt from being logged to httpd log.


Finally we have learned how to completely disable Apache logging if logging is handled by other external application.
 

How to Recover deleted /var/lib/dpkg directory on Debian / Ubuntu Linux server


October 6th, 2021

how-to-recover-accidently-deleted-var-lib-dpkg-directory-on-debian-gnu-linux

Sometimes you might do something stupid, in the hurry like running the wrong rm  command and ending up deleting /var/lib/dpkg on your Debian / Ubuntu system.

by either wrongly issuing the rm to a directory or mistyping rm -r /var/lib/dpkg.
I know this is pretty dumb but sometimes we're all dumb, if you do so and you try to do the regular

root@debian:/ # apt update && apt upgrade

or try to install some random package onwards you will end up with error message:

E: Could not open lock file /var/lib/dpkg/lock – open (2: No such file or directory)

Ending up with this error, does totally blocks your further system administration activities with both apt / aptitude / apt-get as well as with dpkg package management tool.

 

1. The /var/backups recovery directory

Thankfully, by Gods mercy some of Debian Linux system architects has foreseen such issues might occur and have integrated into it the automatic periodic creation of some important files into directory /var/backups/

Hence the next step is to check what kind of backups are available, there:

root@debian:/ # ls -al /var/backups/
total 19892
drwxr-xr-x  7 root root      4096 Sep 24 06:25 ./
drwxr-xr-x 22 root root      4096 Dec 21  2020 ../
-rw-r–r–  1 root root    245760 Aug 20 06:25 alternatives.tar.0
-rw-r–r–  1 root root     15910 Aug 14 06:25 alternatives.tar.1.gz
-rw-r–r–  1 root root     15914 May 29 06:25 alternatives.tar.2.gz
-rw-r–r–  1 root root     15783 Jan 29  2021 alternatives.tar.3.gz
-rw-r–r–  1 root root     15825 Nov 20  2020 alternatives.tar.4.gz
-rw-r–r–  1 root root     15778 Jul 16  2020 alternatives.tar.5.gz
-rw-r–r–  1 root root     15799 Jul  4  2020 alternatives.tar.6.gz
-rw-r–r–  1 root root     80417 Aug 19 14:48 apt.extended_states.0
-rw-r–r–  1 root root      8693 Apr 27 22:40 apt.extended_states.1.gz
-rw-r–r–  1 root root      8658 Apr 17 19:45 apt.extended_states.2.gz
-rw-r–r–  1 root root      8601 Apr 15 00:52 apt.extended_states.3.gz
-rw-r–r–  1 root root      8599 Apr  9 00:26 apt.extended_states.4.gz
-rw-r–r–  1 root root      8542 Mar 18  2021 apt.extended_states.5.gz
-rw-r–r–  1 root root      8549 Mar 18  2021 apt.extended_states.6.gz
-rw-r–r–  1 root root   9030483 Jul  4  2020 aptitude.pkgstates.0
-rw-r–r–  1 root root    628958 May  7  2019 aptitude.pkgstates.1.gz
-rw-r–r–  1 root root    534758 Oct 21  2017 aptitude.pkgstates.2.gz
-rw-r–r–  1 root root    503877 Oct 19  2017 aptitude.pkgstates.3.gz
-rw-r–r–  1 root root    423277 Oct 15  2017 aptitude.pkgstates.4.gz
-rw-r–r–  1 root root    420899 Oct 14  2017 aptitude.pkgstates.5.gz
-rw-r–r–  1 root root    229508 May  5  2015 aptitude.pkgstates.6.gz
-rw-r–r–  1 root root        11 Oct 14  2017 dpkg.arch.0
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.1.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.2.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.3.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.4.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.5.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.6.gz
-rw-r–r–  1 root root      1319 Apr 27 22:28 dpkg.diversions.0
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.1.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.2.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.3.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.4.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.5.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.6.gz
-rw-r–r–  1 root root       375 Aug 23  2018 dpkg.statoverride.0
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.1.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.2.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.3.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.4.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.5.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.6.gz
-rw-r–r–  1 root root   3363749 Sep 23 14:32 dpkg.status.0
-rw-r–r–  1 root root    763524 Aug 19 14:48 dpkg.status.1.gz
-rw-r–r–  1 root root    760198 Aug 17 19:41 dpkg.status.2.gz
-rw-r–r–  1 root root    760176 Aug 13 12:48 dpkg.status.3.gz
-rw-r–r–  1 root root    760105 Jul 16 15:25 dpkg.status.4.gz
-rw-r–r–  1 root root    759807 Jun 28 15:18 dpkg.status.5.gz
-rw-r–r–  1 root root    759554 May 28 16:22 dpkg.status.6.gz

drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T00:22:30.p1e5J8/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T00:24:02.dAUgDs/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T12:29:51.FX27WJ/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T21:18:26.bPQWlW/
drwx——  2 root root      4096 Jul 16  2019 ejabberd-2019-07-16T00:49:52.Gy3sus/
-rw——-  1 root root      2512 Oct 20  2020 group.bak
-rw——-  1 root shadow    1415 Oct 20  2020 gshadow.bak
-rw——-  1 root root      7395 May 11 22:56 passwd.bak
-rw——-  1 root shadow    7476 May 11 22:56 shadow.bak

Considering the situation the important files for us that could, help us restore our previous list of packages, we had installed on the Debian are files under /var/backups/dpkg.status*

Luckily debian based systems keeps backups of its important files that can be used later on for system recovery activities.
Below is a common structure of /var/lib/dpkg on a deb based system.

hipo@debian:/home/hipo$ ls -l /var/lib/dpkg/
total 11504
drwxr-xr-x 2 root root    4096 Aug 19 14:33 alternatives/
-rw-r–r– 1 root root      11 Oct 14  2017 arch
-rw-r–r– 1 root root 2199402 Oct 19  2017 available
-rw-r–r– 1 root root 2197483 Oct 19  2017 available-old
-rw-r–r– 1 root root       8 Sep  6  2012 cmethopt
-rw-r–r– 1 root root    1319 Apr 27 22:28 diversions
-rw-r–r– 1 root root    1266 Nov 18  2020 diversions-old
drwxr-xr-x 2 root root  606208 Sep 23 14:32 info/
-rw-r—– 1 root root       0 Sep 23 14:32 lock
-rw-r—– 1 root root       0 Mar 18  2021 lock-frontend
drwxr-xr-x 2 root root    4096 Sep 17  2012 parts/
-rw-r–r– 1 root root     375 Aug 23  2018 statoverride
-rw-r–r– 1 root root     337 Aug 13  2018 statoverride-old
-rw-r–r– 1 root root 3363749 Sep 23 14:32 status
-rw-r–r– 1 root root 3363788 Sep 23 14:32 status-old
drwxr-xr-x 2 root root    4096 Aug 19 14:48 triggers/
drwxr-xr-x 2 root root    4096 Sep 23 14:32 updates/

 

2. Recreate basic /var/lib/dpkg directory and files structures

As you can see, there are 5 directories and the status file and some other files. 
Hence the first step is to restore the lost directory structure.

hipo@debian: ~$ sudo mkdir -p /var/lib/dpkg/{alternatives,info,parts,triggers,updates}


3. Recover /var/lib/dpkg/status file

Further on recover the dpkg status file from backup

hipo@debian: ~$  sudo cp /var/backups/dpkg.status.0 /var/lib/dpkg/status


4. Check dpkg package installation works again and reinstall base-files

Next check if dpkg – debian package manager is now working, by simply trying to download dpkg*.deb reinstalling it.

root@debian:/root # apt-get download dpkg
# sudo dpkg -i dpkg*.deb

If you get no errors next step is to reinstall base-files which is important package on which dpkg depends.

root@debian:/root # apt-get download base-files

root@debian:/root # sudo dpkg -i base-files*.deb

 

5. Update deb system package list and db consistency

Onwards try to update system package list and check dpkg / apt database consistency.

root@debian:/root # dpkg –audit

root@debian:/root # sudo apt-get update

root@debian:/root # sudo apt-get check


The result should be more of the files in /var/lib/dpkg should appear, thus list the directory again and compare to the earlier given list of it, they should be similar.

root@debian:/root # ls -l /var/lib/dpkg

6. Reinstall completely from source code dpkg, if nothing else works
 

If some files are missing they should get created with a normal daily sysadmin package management tasks so no worries.

In case if after attempting to upgrade the system or install a package with apt, you get some nasty error like:
 

'/usr/local/var/lib/dpkg/status' for reading: No such file or directory


Then the next and final thing to try as a recovery is to download compile from a new and reinstall dpkg from source code!

 

root@debian:/ # wget https://launchpad.net/ubuntu/+archive/primary/+files/dpkg_1.16.1.2ubuntu7.2.tar.bz2
root@debian:/ # tar -xvf dpkg_1.16*

root@debian:/ # cd dpkg-1.16*

root@debian:/ # ./configure

root@debian:/ # make

root@debian:/ # make install


Hopefully you'll have gcc and development tools provided by build-essential .deb package otherwise you have to download and compile this ones as well 🙂
If this doesn't bring you back the installed packages you had priorly (hopefully not), then waste no more time and do a backup of the main things on the server, and reinstall it completely.

The moral out of this incident is always to implement always to your system a good back up system and regularly create backups of /var/lib/dpkg , /etc/ , /usr/local* and other important files on a remote backup server, to be able to easily recover if you do by mistake something whacky.
 

Hope that helped anyone. Cheers 🙂

Install and enable Sysstats IO / DIsk / CPU / Network monitoring console suite on Redhat 8.3, Few sar useful command examples


September 28th, 2021

linux-sysstat-monitoring-logo

 

Why to monitoring CPU, Memory, Hard Disk, Network usage etc. with sysstats tools?
 

Using system monitoring tools such as Zabbix, Nagios Monit is a good approach, however sometimes due to zabbix server interruptions you might not be able to track certain aspects of system performance on time. Thus it is always a good idea to 
Gain more insights on system peroformance from command line. Of course there is cmd tools such as iostat and top, free, vnstat that provides plenty of useful info on system performance issues or bottlenecks. However from my experience to have a better historical data that is systimized and all the time accessible from console it is a great thing to have sysstat package at place. Since many years mostly on every server I administer, I've been using sysstats to monitor what is going on servers over a short time frames and I'm quite happy with it. In current company we're using Redhats and CentOS-es and I had to install sysstats on Redhat 8.3. I've earlier done it multiple times on Debian / Ubuntu Linux and while I've faced on some .deb distributions complications of making sysstat collect statistics I've come with an article on Howto fix sysstat Cannot open /var/log/sysstat/sa no such file or directory” on Debian / Ubuntu Linux
 

Sysstat contains the following tools related to collecting I/O and CPU statistics:
iostat
Displays an overview of CPU utilization, along with I/O statistics for one or more disk drives.
mpstat
Displays more in-depth CPU statistics.
Sysstat also contains tools that collect system resource utilization data and create daily reports based on that data. These tools are:
sadc
Known as the system activity data collector, sadc collects system resource utilization information and writes it to a file.
sar
Producing reports from the files created by sadc, sar reports can be generated interactively or written to a file for more intensive analysis.

My experience with CentOS 7 and Fedora to install sysstat it was pretty straight forward, I just had to install it via yum install sysstat wait for some time and use sar (System Activity Reporter) tool to report collected system activity info stats over time.
Unfortunately it seems on RedHat 8.3 as well as on CentOS 8.XX instaling sysstats does not work out of the box.

To complete a successful installation of it on RHEL 8.3, I had to:

[root@server ~]# yum install -y sysstat


To make sysstat enabled on the system and make it run, I've enabled it in sysstat

[root@server ~]# systemctl enable sysstat


Running immediately sar command, I've faced the shitty error:


Cannot open /var/log/sysstat/sa18:
No such file or directory. Please check if data collecting is enabled”

 

Once installed I've waited for about 5 minutes hoping, that somehow automatically sysstat would manage it but it didn't.

To solve it, I've had to create additionally file /etc/cron.d/sysstat (weirdly RPM's post install instructions does not tell it to automatically create it)

[root@server ~]# vim /etc/cron.d/sysstat

# run system activity accounting tool every 10 minutes
0 * * * * root /usr/lib64/sa/sa1 60 59 &
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A &

 

  • /usr/local/lib/sa1 is a shell script that we can use for scheduling cron which will create daily binary log file.
  • /usr/local/lib/sa2 is a shell script will change binary log file to human-readable form.

 

[root@server ~]# chmod 600 /etc/cron.d/sysstat

[root@server ~]# systemctl restart sysstat


In a while if sysstat is working correctly you should get produced its data history logs inside /var/log/sa

[root@server ~]# ls -al /var/log/sa 


Note that the standard sysstat history files on Debian and other modern .deb based distros such as Debian 10 (in  y.2021) is stored under /var/log/sysstat

Here is few useful uses of sysstat cmds


1. Check with sysstat machine history SWAP and RAM Memory use


To lets say check last 10 minutes SWAP memory use:

[hipo@server yum.repos.d] $ sar -W  |last -n 10
 

Linux 4.18.0-240.el8.x86_64 (server)       09/28/2021      _x86_64_        (8 CPU)

12:00:00 AM  pswpin/s pswpout/s
12:00:01 AM      0.00      0.00
12:01:01 AM      0.00      0.00
12:02:01 AM      0.00      0.00
12:03:01 AM      0.00      0.00
12:04:01 AM      0.00      0.00
12:05:01 AM      0.00      0.00
12:06:01 AM      0.00      0.00

[root@ccnrlb01 ~]# sar -r | tail -n 10
14:00:01        93008   1788832     95.06         0   1357700    725740      9.02    795168    683484        32
14:10:01        78756   1803084     95.81         0   1358780    725740      9.02    827660    652248        16
14:20:01        92844   1788996     95.07         0   1344332    725740      9.02    813912    651620        28
14:30:01        92408   1789432     95.09         0   1344612    725740      9.02    816392    649544        24
14:40:01        91740   1790100     95.12         0   1344876    725740      9.02    816948    649436        36
14:50:01        91688   1790152     95.13         0   1345144    725740      9.02    817136    649448        36
15:00:02        91544   1790296     95.14         0   1345448    725740      9.02    817472    649448        36
15:10:01        91108   1790732     95.16         0   1345724    725740      9.02    817732    649340        36
15:20:01        90844   1790996     95.17         0   1346000    725740      9.02    818016    649332        28
Average:        93473   1788367     95.03         0   1369583    725074      9.02    800965    671266        29

 

2. Check system load? Are my processes waiting too long to run on the CPU?

[root@server ~ ]# sar -q |head -n 10
Linux 4.18.0-240.el8.x86_64 (server)       09/28/2021      _x86_64_        (8 CPU)

12:00:00 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:00:01 AM         0       272      0.00      0.02      0.00         0
12:01:01 AM         1       271      0.00      0.02      0.00         0
12:02:01 AM         0       268      0.00      0.01      0.00         0
12:03:01 AM         0       268      0.00      0.00      0.00         0
12:04:01 AM         1       271      0.00      0.00      0.00         0
12:05:01 AM         1       271      0.00      0.00      0.00         0
12:06:01 AM         1       265      0.00      0.00      0.00         0


3. Show various CPU statistics per CPU use
 

On a multiprocessor, multi core server sometimes for scripting it is useful to fetch processor per use historic data, 
this can be attained with:

 

[hipo@server ~ ] $ mpstat -P ALL
Linux 4.18.0-240.el8.x86_64 (server)       09/28/2021      _x86_64_        (8 CPU)

06:08:38 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
06:08:38 PM  all    0.17    0.02    0.25    0.00    0.05    0.02    0.00    0.00    0.00   99.49
06:08:38 PM    0    0.22    0.02    0.28    0.00    0.06    0.03    0.00    0.00    0.00   99.39
06:08:38 PM    1    0.28    0.02    0.36    0.00    0.08    0.02    0.00    0.00    0.00   99.23
06:08:38 PM    2    0.27    0.02    0.31    0.00    0.06    0.01    0.00    0.00    0.00   99.33
06:08:38 PM    3    0.15    0.02    0.22    0.00    0.03    0.01    0.00    0.00    0.00   99.57
06:08:38 PM    4    0.13    0.02    0.20    0.01    0.03    0.01    0.00    0.00    0.00   99.60
06:08:38 PM    5    0.14    0.02    0.27    0.00    0.04    0.06    0.01    0.00    0.00   99.47
06:08:38 PM    6    0.10    0.02    0.17    0.00    0.04    0.02    0.00    0.00    0.00   99.65
06:08:38 PM    7    0.09    0.02    0.15    0.00    0.02    0.01    0.00    0.00    0.00   99.70


 

sar-sysstat-cpu-statistics-screenshot

Monitor processes and threads currently being managed by the Linux kernel.

[hipo@server ~ ] $ pidstat

pidstat-various-random-process-statistics

[hipo@server ~ ] $ pidstat -d 2


pidstat-show-processes-with-most-io-activities-linux-screenshot

This report tells us that there is few processes with heave I/O use Filesystem system journalling daemon jbd2, apache, mysqld and supervise, in 3rd column you see their respective PID IDs.

To show threads used inside a process (like if you press SHIFT + H) inside Linux top command:

[hipo@server ~ ] $ pidstat -t -p 10765 1 3

Linux 4.19.0-14-amd64 (server)     28.09.2021     _x86_64_    (10 CPU)

21:41:22      UID      TGID       TID    %usr %system  %guest   %wait    %CPU   CPU  Command
21:41:23      108     10765         –    1,98    0,99    0,00    0,00    2,97     1  mysqld
21:41:23      108         –     10765    0,00    0,00    0,00    0,00    0,00     1  |__mysqld
21:41:23      108         –     10768    0,00    0,00    0,00    0,00    0,00     0  |__mysqld
21:41:23      108         –     10771    0,00    0,00    0,00    0,00    0,00     5  |__mysqld
21:41:23      108         –     10784    0,00    0,00    0,00    0,00    0,00     7  |__mysqld
21:41:23      108         –     10785    0,00    0,00    0,00    0,00    0,00     6  |__mysqld
21:41:23      108         –     10786    0,00    0,00    0,00    0,00    0,00     2  |__mysqld

10765 – is the Process ID whose threads you would like to list

With pidstat, you can further monitor processes for memory leaks with:

[hipo@server ~ ] $ pidstat -r 2

 

4. Report paging statistics for some old period

 

[root@server ~ ]# sar -B -f /var/log/sa/sa27 |head -n 10
Linux 4.18.0-240.el8.x86_64 (server)       09/27/2021      _x86_64_        (8 CPU)

15:42:26     LINUX RESTART      (8 CPU)

15:55:30     LINUX RESTART      (8 CPU)

04:00:01 PM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
04:01:01 PM      0.00     14.47    629.17      0.00    502.53      0.00      0.00      0.00      0.00
04:02:01 PM      0.00     13.07    553.75      0.00    419.98      0.00      0.00      0.00      0.00
04:03:01 PM      0.00     11.67    548.13      0.00    411.80      0.00      0.00      0.00      0.00

 

5.  Monitor Received RX and Transmitted TX network traffic perl Network interface real time
 

To print out Received and Send traffic per network interface 4 times in a raw

sar-sysstats-network-traffic-statistics-screenshot
 

[hipo@server ~ ] $ sar -n DEV 1 4


To continusly monitor all network interfaces I/O traffic

[hipo@server ~ ] $ sar -n DEV 1


To only monitor a certain network interface lets say loopback interface (127.0.0.1) received / transmitted bytes

[hipo@server yum.repos.d] $  sar -n DEV 1 2|grep -i lo
06:29:53 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:29:54 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00


6. Monitor block devices use
 

To check block devices use 3 times in a raw
 

[hipo@server yum.repos.d] $ sar -d 1 3


sar-sysstats-blockdevice-statistics-screenshot
 

7. Output server monitoring data in CSV database structured format


For preparing a nice graphs with Excel from CSV strucuted file format, you can dump the collected data as so:

 [root@server yum.repos.d]# sadf -d /var/log/sa/sa27 — -n DEV | grep -v lo|head -n 10
server-name-fqdn;-1;2021-09-27 13:42:26 UTC;LINUX-RESTART    (8 CPU)
# hostname;interval;timestamp;IFACE;rxpck/s;txpck/s;rxkB/s;txkB/s;rxcmp/s;txcmp/s;rxmcst/s;%ifutil
server-name-fqdn;-1;2021-09-27 13:55:30 UTC;LINUX-RESTART    (8 CPU)
# hostname;interval;timestamp;IFACE;rxpck/s;txpck/s;rxkB/s;txkB/s;rxcmp/s;txcmp/s;rxmcst/s;%ifutil
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth1;19.42;16.12;1.94;1.68;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth0;7.18;9.65;0.55;0.78;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth2;5.65;5.13;0.42;0.39;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth1;18.90;15.55;1.89;1.60;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth0;7.15;9.63;0.55;0.74;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth2;5.67;5.15;0.42;0.39;0.00;0.00;0.00;0.00

To graph the output data you can use Excel / LibreOffice's Excel equivalent Calc or if you need to dump a CSV sar output and generate it on the fly from a script  use gnuplot 


What we've learned?


How to install and enable on cron sysstats on Redhat and CentOS 8 Linux ? 
How to continuously monitor CPU / Disk and Network, block devices, paging use and processes and threads used by the kernel per process ?  
As well as how to export previously collected data to CSV to import to database or for later use inrder to generate graphic presentation of data.
Cheers ! 🙂

 

How to redirect TCP port traffic from Internet Public IP host to remote local LAN server, Redirect traffic for Apache Webserver, MySQL, or other TCP service to remote host


September 23rd, 2021

 

 

Linux-redirect-forward-tcp-ip-port-traffic-from-internet-to-remote-internet-LAN-IP-server-rinetd-iptables-redir

 

 

1. Use the good old times rinetd – internet “redirection server” service


Perhaps, many people who are younger wouldn't remember rinetd's use was pretty common on old Linuxes in the age where iptables was not on the scene and its predecessor ipchains was so common.
In the raise of mass internet rinetd started loosing its popularity because the service was exposed to the outer world and due to security holes and many exploits circulating the script kiddie communities
many servers get hacked "pwned" in the jargon of the script kiddies.

rinetd is still available even in modern Linuxes and over the last years I did not heard any severe security concerns regarding it, but the old paranoia perhaps and the set to oblivion makes it still unpopular soluttion for port redirect today in year 2021.
However for a local secured DMZ lans I can tell you that its use is mostly useful and I chooes to use it myself, everynow and then due to its simplicity to configure and use.
rinetd is pretty standard among unixes and is also available in old Sun OS / Solaris and BSD-es and pretty much everything on the Unix scene.

Below is excerpt from 'man rinetd':

 

DESCRIPTION
     rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs
     specified in the file /etc/rinetd.conf.  Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe im‐
     pact on the machine. This makes it practical to run TCP services on machines inside an IP masquerading firewall. rinetd does not redirect FTP, because FTP requires more than
     one socket.
     rinetd is typically launched at boot time, using the following syntax:      /usr/sbin/rinetd      The configuration file is found in the file /etc/rinetd.conf, unless another file is specified using the -c command line option.

To use rinetd on any LInux distro you have to install and enable it with apt or yum as usual. For example on my Debian GNU / Linux home machine to use it I had to install .deb package, enable and start it it via systemd :

 

server:~# apt install –yes rinetd

server:~#  systemctl enable rinetd


server:~#  systemctl start rinetd


server:~#  systemctl status rinetd
● rinetd.service
   Loaded: loaded (/etc/init.d/rinetd; generated)
   Active: active (running) since Tue 2021-09-21 10:48:20 EEST; 2 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 1 (limit: 4915)
   Memory: 892.0K
   CGroup: /system.slice/rinetd.service
           └─1364 /usr/sbin/rinetd


rinetd is doing the traffic redirect via a separate process daemon, in order for it to function once you have service up check daemon is up as well.

root@server:/home/hipo# ps -ef|grep -i rinet
root       359     1  0 16:10 ?        00:00:00 /usr/sbin/rinetd
root       824 26430  0 16:10 pts/0    00:00:00 grep -i rinet

+ Configuring a new port redirect with rinetd

 

Is pretty straight forward everything is handled via one single configuration – /etc/rinetd.conf

The format (syntax) of a forwarding rule is as follows:

     [bindaddress] [bindport] [connectaddress] [connectport]


Besides that rinetd , could be used as a primitive firewall substitute to iptables, general syntax of allow deny an IP address is done with (allow, deny) keywords:
 

allow 192.168.2.*
deny 192.168.2.1?


To enable logging to external file ,you'll have to include in the configuration:

# logging information
logfile /var/log/rinetd.log

Here is an example rinetd.conf configuration, redirecting tcp mysql 3306, nginx on port 80 and a second web service frontend for ILO to server reachable via port 8888 and a redirect from External IP to local IP SMTP server.

 

#
# this is the configuration file for rinetd, the internet redirection server
#
# you may specify global allow and deny rules here
# only ip addresses are matched, hostnames cannot be specified here
# the wildcards you may use are * and ?
#
# allow 192.168.2.*
# deny 192.168.2.1?


#
# forwarding rules come here
#
# you may specify allow and deny rules after a specific forwarding rule
# to apply to only that forwarding rule
#
# bindadress    bindport  connectaddress  connectport


# logging information
logfile /var/log/rinetd.log
83.228.93.76        80            192.168.0.20       80
192.168.0.2        3306            192.168.0.19        3306
83.228.93.76        443            192.168.0.20       443
# enable for access to ILO
83.228.93.76        8888            192.168.1.25 443

127.0.0.1    25    192.168.0.19    25


83.228.93.76 is my external ( Public )  IP internet address where 192.168.0.20, 192.168.0.19, 192.168.0.20 (are the DMZ-ed Lan internal IPs) with various services.

To identify the services for which rinetd is properly configured to redirect / forward traffic you can see it with netstat or the newer ss command
 

root@server:/home/hipo# netstat -tap|grep -i rinet
tcp        0      0 www.pc-freak.net:8888   0.0.0.0:*               LISTEN      13511/rinetd      
tcp        0      0 www.pc-freak.n:http-alt 0.0.0.0:*               LISTEN      21176/rinetd        
tcp        0      0 www.pc-freak.net:443   0.0.0.0:*               LISTEN      21176/rinetd      

 

+ Using rinetd to redirect External interface IP to loopback's port (127.0.0.1)

 

If you have the need to redirect an External connectable living service be it apache mysql / privoxy / squid or whatever rinetd is perhaps the tool of choice (especially since there is no way to do it with iptables.

If you want to redirect all traffic which is accessed via Linux's loopback interface (localhost) to be reaching a remote host 11.5.8.1 on TCP port 1083 and 1888, use below config

# bindadress    bindport  connectaddress  connectport
11.5.8.1        1083            127.0.0.1       1083
11.5.8.1        1888            127.0.0.1       1888

 

For a quick and dirty solution to redirect traffic rinetd is very useful, however you'll have to keep in mind that if you want to redirect traffic for tens of thousands of connections constantly originating from the internet you might end up with some disconnects as well as notice a increased use of rinetd CPU use with the incrased number of forwarded connections.

 

2. Redirect TCP / IP port using DNAT iptables firewall rules

 

Lets say you have some proxy, webservice or whatever service running on port 5900 to be redirected with iptables.
The easeiest legacy way is to simply add the redirection rules to /etc/rc.local​. In newer Linuxes rc.local so if you decide to use,
you'll have to enable rc.local , I've written earlier a short article on how to enable rc.local on newer Debian, Fedora, CentOS

 

# redirect 5900 TCP service 
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -I PREROUTING -p tcp –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -I OUTPUT -p tcp -o lo –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 5900 -j DNAT  –to-destination 192.168.1.8:5900
iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 5900 -j REDIRECT –to-ports 5900

 

Here is another two example which redirects port 2208 (which has configured a bind listener for SSH on Internal host 192.168.0.209:2208) from External Internet IP address (XXX.YYY.ZZZ.XYZ) 
 

# Port redirect for SSH to VM on openxen internal Local lan server 192.168.0.209 
-A PREROUTING  -p tcp –dport 2208 -j DNAT –to-destination 192.168.0.209:2208
-A POSTROUTING -p tcp –dst 192.168.0.209 –dport 2208 -j SNAT –to-source 83.228.93.76

 

3. Redirect TCP traffic connections with redir tool

 

If you look for an easy straight forward way to redirect TCP traffic, installing and using redir (ready compiled program) might be a good idea.


root@server:~# apt-cache show redir|grep -i desc -A5 -B5
Version: 3.2-1
Installed-Size: 60
Maintainer: Lucas Kanashiro <kanashiro@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.15)
Description-en: Redirect TCP connections
 It can run under inetd or stand alone (in which case it handles multiple
 connections).  It is 8 bit clean, not limited to line mode, is small and
 light. Supports transparency, FTP redirects, http proxying, NAT and bandwidth
 limiting.
 .
 redir is all you need to redirect traffic across firewalls that authenticate
 based on an IP address etc. No need for the firewall toolkit. The
 functionality of inetd/tcpd and "redir" will allow you to do everything you
 need without screwy telnet/ftp etc gateways. (I assume you are running IP
 Masquerading of course.)

Description-md5: 2089a3403d126a5a0bcf29b22b68406d
Homepage: https://github.com/troglobit/redir
Tag: interface::daemon, network::server, network::service, role::program,
 use::proxying
Section: net
Priority: optional

 

 

server:~# apt-get install –yes redir

Here is a short description taken from its man page 'man redir'

 

DESCRIPTION
     redir redirects TCP connections coming in on a local port, [SRC]:PORT, to a specified address/port combination, [DST]:PORT.  Both the SRC and DST arguments can be left out,
     redir will then use 0.0.0.0.

     redir can be run either from inetd or as a standalone daemon.  In –inetd mode the listening SRC:PORT combo is handled by another process, usually inetd, and a connected
     socket is handed over to redir via stdin.  Hence only [DST]:PORT is required in –inetd mode.  In standalone mode redir can run either in the foreground, -n, or in the back‐
     ground, detached like a proper UNIX daemon.  This is the default.  When running in the foreground log messages are also printed to stderr, unless the -s flag is given.

     Depending on how redir was compiled, not all options may be available.

 

+ Use redir to redirect TCP traffic one time

 

Lets say you have a MySQL running on remote machine on some internal or external IP address, lets say 192.168.0.200 and you want to redirect all traffic from remote host to the machine (192.168.0.50), where you run your Apache Webserver, which you want to configure to use
as MySQL localhost TCP port 3306.

Assuming there are no irewall restrictions between Host A (192.168.0.50) and Host B (192.168.0.200) is already permitting connectivity on TCP/IP port 3306 between the two machines.

To open redirection from localhost on 192.168.0.50 -> 192.168.0.200:

 

server:~# redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306

 

If you need other third party hosts to be additionally reaching 192.168.0.200 via 192.168.0.50 TCP 3306.

root@server:~# redir –laddr=192.168.0.50 –lport=3306 –caddr=192.168.0.200 –cport=3306


Of course once you close, the /dev/tty or /dev/vty console the connection redirect will be cancelled.

 

+ Making TCP port forwarding from Host A to Host B permanent


One solution to make the redir setup rules permanent is to use –rinetd option or simply background the process, nevertheless I prefer to use instead GNU Screen.
If you don't know screen is a vVrtual Console Emulation manager with VT100/ANSI terminal emulation to so, if you don't have screen present on the host install it with whatever Linux OS package manager is present and run:

 

root@server:~#screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306'

 

That would run it into screen session and detach so you can later connect, if you want you can make redir to also log connections via syslog with ( -s) option.

I found also useful to be able to track real time what's going on currently with the opened redirect socket by changing redir log level.

Accepted log level is:

 

  -l, –loglevel=LEVEL
             Set log level: none, err, notice, info, debug.  Default is notice.

 

root@server:/ # screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3308 –caddr=192.168.0.200 –cport=3306 -l debug'

 

To test connectivity works as expected use telnet:
 

root@server:/ # telnet localhost 3308
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
g
5.5.5-10.3.29-MariaDB-0+deb10u1-log�+c2nWG>B���o+#ly=bT^]79mysql_native_password

6#HY000Proxy header is not accepted from 192.168.0.19 Connection closed by foreign host.

once you attach to screen session with

 

root@server:/home #  screen -r

 

You will get connectivity attempt from localhost logged : .
 

redir[10640]: listening on 127.0.0.1:3306
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10793]: peer IP is 127.0.0.1
redir[10793]: peer socket is 25592
redir[10793]: target IP address is 192.168.0.200
redir[10793]: target port is 3306
redir[10793]: Connecting 127.0.0.1:25592 to 127.0.0.1:3306
redir[10793]: Entering copyloop() – timeout is 0
redir[10793]: Disconnect after 1 sec, 165 bytes in, 4 bytes out

The downsides of using redir is redirection is handled by the separate process which is all time hanging in the process list, as well as the connection redirection speed of incoming connections might be about at least 30% slower to if you simply use a software (firewall ) redirect such as iptables. If you use something like kernel IP set ( ipsets ). If you hear of ipset for a first time and you wander whta it is below is short package description.

 

root@server:/root# apt-cache show ipset|grep -i description -A13 -B5
Maintainer: Debian Netfilter Packaging Team <pkg-netfilter-team@lists.alioth.debian.org>
Architecture: amd64
Provides: ipset-6.38
Depends: iptables, libc6 (>= 2.4), libipset11 (>= 6.38-1~)
Breaks: xtables-addons-common (<< 1.41~)
Description-en: administration tool for kernel IP sets
 IP sets are a framework inside the Linux 2.4.x and 2.6.x kernel which can be
 administered by the ipset(8) utility. Depending on the type, currently an
 IP set may store IP addresses, (TCP/UDP) port numbers or IP addresses with
 MAC addresses in a  way which ensures lightning speed when matching an
 entry against a set.
 .
 If you want to
 .
  * store multiple IP addresses or port numbers and match against the
    entire collection using a single iptables rule.
  * dynamically update iptables rules against IP addresses or ports without
    performance penalty.
  * express complex IP address and ports based rulesets with a single
    iptables rule and benefit from the speed of IP sets.

 .
 then IP sets may be the proper tool for you.
Description-md5: d87e199641d9d6fbb0e52a65cf412bde
Homepage: http://ipset.netfilter.org/
Tag: implemented-in::c, role::program
Section: net
Priority: optional
Filename: pool/main/i/ipset/ipset_6.38-1.2_amd64.deb
Size: 50684
MD5sum: 095760c5db23552a9ae180bd58bc8efb
SHA256: 2e2d1c3d494fe32755324bf040ffcb614cf180327736c22168b4ddf51d462522

Change Windows 10 default lock screen image via win registry LockScreenImage key change


September 21st, 2021

fix-lock-screen-missing-change-option-on-windows-10-windows-registry-icon

If you do work for a corporation on a Windows machine that is part of Windows Active Directory domain or a Microsoft 365 environment and your Domain admimistrator after some of the scheduled updates. Has enforced a Windows lock screen image change.
You  might be surprised to have some annoying corporation logo picture shown as a default Lock Screen image on your computer on next reoboot. Perhaps for some people it doesn't matter but for as a person who seriously like customizations, and a valuer of
freedom having an enforced picture logo each time I press CTRL + L (To lock my computer) is really annoying.

The logical question hence was how to reverse my desired image as  a default lock screen to enkoy. Some would enjoy some relaxing picture of a Woods, Cave or whatever Natural place landscape. I personally prefer simplicity so I simply use a simple purely black
background.

To do it you'll have anyways to have some kind of superuser access to the computer. At the company I'm epmloyeed, it is possible to temporary request Administrator access this is done via a software installed on the machine. So once I request it I become
Administratof of machine for 20 minutes. In that time I do used a 'Run as Administartor' command prompt cmd.exe and inside Windows registry do the following Registry change.

The first logical thing to do is to try to manually set the picture via:
 

Settings ->  Lock Screen

But unfortunately as you can see in below screenshot, there was no way to change the LockScreen background image.

Windows-settings-lockscreen-screenshot

In Windows Registry Editor

I had to go to registry path


[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\.]

And from there in create a new "String Value" key
 

"LockScreenImage"


so full registry key path should be equal to:


[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Personalization\LockScreenImage]"

The value to set is:

C:\Users\a768839\Desktop\var-stuff\background\Desired-background-picture.jpg

windows-registry-change-lock-screen-background-picture-from-registry-screenshot

If you want to set a black background picture for LockScreen like me you can download my black background picture from here.

That's all press CTRL + L  key combination and the black screen background lock screen picture will appear !

Hopefully the Domain admin, would not soon enforce some policty to update the registry keys or return your old registry database from backup if something crashs out with something strange to break just set configuration.

To test whether the setting will stay permanent after the next scheduled Windows PC update of policies enforced by the Active Directory (AD) sysadmin, run manually from CMD.EXE

C:\> gpupdate /force


The command will download latest policies from Windows Domain, try to lock the screen once again with Control + L, if the background picture is still there most likely the Picture change would stay for a long.
If you get again the corporation preset domain background instead,  you're out of luck and will have to follow the same steps every, now and then after a domani policy update.

Enjoy your new smooth LockScreen Image 🙂

 

How to create SD Card DATA dump image to .ISO with dd and mount it with imdisk from command line on Windows CygWin with MobaXterm


September 18th, 2021

dd-command-logo
I'm forced to use Windows every now and then and do some ordinary things which I do usually on Linux such as dumping the content of my Android phone SD Card SanDisk, Kingston etc. to .ISO image etc.

On Linux creating and mounting a data copy of a whole SD Card is a relatively simple thing and there are plenty of ways to do it such as using the dd ( command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files as said in the command manual .- e.g. ''man dd'. ). On Microsoft Windows environment perhaps one of easiest ways is to use WinCDEmu (which is relatively free under LGPL License).
WinCDEmu is capable of doing plenty of things such as:
 

  • One-click mounting of ISO, CUE, NRG, MDS/MDF, CCD, IMG images.

  • Supports unlimited amount of virtual drives.

  • Runs on 32-bit and 64-bit Windows versions from XP to Windows 10.

  • Allows creating ISO images through a context menu in Explorer.

  • Small installer size – less than 2MB!

  • Have a portable version

WinCDEmu is a nice piece of software that perhaps every Win poweruser can enjoy, plus it has a nice Graphical frontend:

wincdemu-graphical-create-iso-and-mount-so-ms-windows-software

But what if you're a console geek, like me and you end up forced to be using Windows on your Work PC and you still need to create .iso dump of your Mobile SD Card or external attached Hard Drive, without the graphical mambo jumbo in the old fashioned way with dd?

Luckily Windows advanced command lined users could massively benefit from Cygwin + Mobaxterm (if you don't know or used MobaXterm and you still use things like Putty / SuperPutty or SecureCRT – perhaps you can reconsider and make your sysadmin life easier with MobaXerm gnome-terminal like SSH tabbed Windows alternative.

Once having mobaxterm + cygwin you have dd installed on the Windows host as it is part of the busybox minimal environment and you can use it in the same manner as your used in Linux environment.

sdcard-sandisk-drive-my-computer-windows-screenshot
 

1. Using dd to copy files on Linux / UNIX OS with a dialog status bar

To use dd the usual syntax on Linux / BSD / Unix is:
 

dd if=/dev/dev-name_ID of=/path/to/directory/dump/location.iso bs=2048

 

As 2048 BS (Bytes) per second is quite a low value usually on Modern operating systems, this bytesize is usually increased to some MBs  ( Megabytes).

For example if the reading from carrier  is Solid State Drive Disk (SSD) supporting 100 MBs per second and the output SD Card is a 32 Bit Kingston Plus+ drive with whose write speed is up to 50 ~ 100 MBs, you can use cmd as:

dd if=/dev/dev-name_ID of=/path/to/directory/dump/location.iso bs=100M


If you need to have a progress on the dd copy (in case if you copy some large SD Card 128 GB or 256GB or a full copy of a hard drive partition that is really big lets say 8 Terabytes of data, dialog and pv comes quite handy.

To use them install them first:

# apt-get install –yes pv dialog


Next to have a beautiful ncurses dialog box with the status (very useful if you're shell scripting), use:
 

(pv -n /dev/sda | dd of=/dev/sdb bs=128M conv=notrunc,noerror) 2>&1 | dialog –gauge "Running dd command (cloning), please wait…" 10 70 0

pv-dialog-dd-command-ncurses-status-screenshot-gnu-linux
 

2. Listing the avaialble copy drives /dev/sda /dev/sdb1 … etc. disk locations on Windows 7 / 10 / 11 OS

[User.T420-89] ➤ for F in /dev/s* ; do echo "$F    $(cygpath -w $F)" ; done

check-drives-loop-on-cygwin-to-be-used-later-with-dd-copy-iso-creating-imageCheck drives device naming on WIndows PC – Screenshot extract from Mobaxterm

As you can see the drive location we've seen in Windows Explorer is located at drive E: above bash for loop reveals us this is located and readable from CygWin / MobaxTerm at /dev/sdb1


3. Create .iso image file on WIndows OS with dd command
 

To create a full data copy dump of to .iso (image file) with dd on Windows , I had to run:

[User.T420-89] ➤ dd if=/dev/sdb1 of=sdcard-blu-r1-hd-sdcard-backup_10092021a.img bs=100M

75+1 records in
75+1 records out
7944011776 bytes (7.4GB) copied, 391.794316 seconds, 19.3MB/s


dd-copy-drive-data-screenshot-100mb-bitesize-windows-mobaxterm


4. Mount the newly create dd Image with imdisk

In order to test the image is properly created, you can attempt to mount it from command line on Linux, mounting it is quite easy and is up to mounting the just created .img file as a loopback (loop) device, like so: 

# mount -o loop file.iso /mnt/dir

Unfortunately cygwin and mobaxterm's embedded mount command on Win OS does not support the loopback device so to have it you have to install and use some additional program  such as the upmentioned WinCDEmu or if you prefer to do it fully from command line and further on automate the process of creating a dump of images of attached drives out of a multiple computers (lets say belonging to a Windows Active Directory domain). You might install and use something like:


imdisk 

imdisk-gui-interface-ms-windows-screenshot

imdisk handy tool is  created by Olof Lagerkvist. It is free and open-source software, which  will let you mount image files of hard drive, cd-rom or floppy, and create one or several ramdisks with various parameters either from a command line or via its Graphical interface.

To use imdisk download it from its home page on sourceforge extract and install it, pretty much as any other software it has both 32 bit version as a legacy for old computers as well as 64 bit exe installer.
The general command line use of it follows a cmd syntax like:

  • Mounting .iso image files from command line on WIndows host with imdisk


[User.T420-89] ➤ ImDisk.exe -a -f "sdcard-blu-r1-hd-sdcard-backup_10092021.img" -m #:

Where:
 

  • #: – is the actual drive you would like to mount to.
     
  • -a option stands for attach to, it will configure and attach a virtual disk with the parameters specified and attach it to the system.
     
  • -f – is self explanatory, provides the iso image file naming 

If you want to attach the newly created image to lets say  L:\ windows new mapped drive

ImDisk.exe -a -f "sdcard-blu-r1-hd-sdcard-backup_10092021.img" -m l:

  • Unmount mounted .img image with imdisk from cmd line

[User.T420-89] ➤ imdisk.exe -l
\Device\ImDisk0
                                                                                                                              ✘

[User.T420-89] ➤ imdisk.exe -D -m l:
Notifying applications…
Flushing file buffers…
Locking volume…
Failed, forcing dismount…
Removing device…
Removing mountpoint…
Done.

imdisk-detach-attached-drive-mobaxterm-windows-screenshot

 

What we learned ?

What we have learned in this article is how to use Mobaxterm embedded dd Data Convert and Copy command to prepare full image backups of SD card or external drives on Windows OS. Also few alternative ways were entions such as using WinCDEmu free  open source alternative to DaemonTools program to create / mount or convert the image for the GUI lovers. Also for hard core sysadmins as me was shown how to list drives devices attached to the Win PC {/dev/sda,/dev/sdb} etc. and how to copy partition data with dd just like one would do on Linux OS. Finally to test the created image, I've shown you how to use the imdisk free software tool to attach and detach image to a mapped local Windows drive.

Hope this article learned you something new.

Defining multiple short Server Hostname aliases via SSH config files and defining multiple ssh options for it, Use passwordless authentication via public keys


September 16th, 2021

using-ssh-host-acronym-aliases-ssh-client-explained-openssh-logo

In case you have to access multiple servers from your terminal client such as gnome-terminal, kterminal (if on Linux) or something such as mobaxterm + cygwin (if on Windows) with an opens ssh client (ssh command). There is a nifty trick to save time and keyboard typing through creating shortcuts aliases by adding few definitions inside your $HOME/.ssh/config ( ~/.ssh/config ) for your local non root user or even make the configuration system wide (for all existing local /etc/passwd users) via /etc/ssh/ssh_config.
By adding a pseudonym alias for each server it makes sysadmin life much easier as you don't have to type in each time the FQDN (Fully Qualified Domain Name) hostname of remote accessed Linux / Unix / BSD / Mac OS or even Windows sshd ready hosts accessible via remote TCP/IP port 22.


1. Adding local user remote server pointer aliases via ~/.ssh/config


The file ~/.ssh/config is read by the ssh client part of the openssh-client (Linux OS package) on each invokement of the client, and besides defining a pseudonym for the hosts you like to save you time when accessing remote host and hence increase your productivity. Moreover you can also define various other nice options through it to define specifics of remote ssh session for each desired host such as remote host default SSH port (for example if your OpenSSHD is configured to run on non-standard SSH port as lets say 2022 instead of default port TCP 22 for some reason, e.g. security through obscurity etc.).

 

The general syntax of .ssh/config file si simplistic, it goes like this:
 

Host MACHNE_HOSTNAME

SSH_OPTION1 value1
SSH_OPTION1 value1 value2
SSH_OPTION2 value1 value2

 

Host MACHNE_HOSTNAME

SSH_OPTION value
SSH_OPTION1 value1 value2

  • Another understood syntax if you prefer to not have empty whitespaces is to use ( = )
    between the parameter name and values.

Host MACHINE_HOSTNAME
SSH_config=value
SSH_config1=value1 value2

  • All empty lines and lines starting with the hash shebang sign ( # ) would be ignored.
  • All values are case-sensitive, but parameter names are not.

If you have never so far used the $HOME/.ssh/config you would have to create the file and set the proper permissions to it like so:

mkdir -p $HOME/.ssh
chmod 0700 $HOME/.ssh


Below are examples taken from my .ssh/config configuration for all subdomains for my pcfreak.org domain

 

# Ask for password for every subdomain under pc-freak.net for security
Host *.pcfreak.org
user hipopo
passwordauthentication yes
StrictHostKeyChecking no

# ssh public Key authentication automatic login
Host www1.pc-freak.net
user hipopo
Port 22
passwordauthentication no
StrictHostKeyChecking no

UserKnownHostsFile /dev/null

Host haproxy2
    Hostname 213.91.190.233
    User root
    Port 2218
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/haproxy2.pub    
    StrictHostKeyChecking no
    LogLevel INFO     

Host pcfrxenweb
    Hostname 83.228.93.76
    User root
    Port 2218

    PubkeyAuthentication yes
    IdentityFile ~/.ssh/pcfrxenweb.key    
    StrictHostKeyChecking no

Host pcfreak-sf
    Hostname 91.92.15.51
    User root
    Port 2209
    PreferredAuthentications password
    StrictHostKeyChecking no

    Compression yes


As you can see from above configuration the Hostname could be referring either to IP address or to Hostname.

Now to connect to defined IP 91.92.15.51 you can simply refer to its alias

$ ssh pcfreak-sf -v

and you end up into the machine ssh on port 2209 and you will be prompted for a password.

$ ssh pcfrxenweb -v


would lead to IP 83.228.93.76 SSH on Port 2218 and will use the defined public key for a passwordless login and will save you the password typing each time.

Above ssh command is a short alias you can further use instead of every time typing:

$ ssh -i ~/.ssh/pcfrxenweb.key -p 2218 root@83.228.93.76

There is another nifty trick worthy to mention, if you have a defined hostname such as the above config haproxy2 to use a certain variables, but you would like to override some option for example you don't want to connet by default with User root, but some other local account, lets say ssh as devuser@haproxy2 you can type:

$ ssh -o "User=dev" devuser

StrictHostKeyChecking no

– variable will instruct the ssh to not check if the finger print of remote host has changed. Usually this finger print check sum changes in case if for example for some reason the opensshd gets updated or the default /etc/ssh/ssh_host_dsa_key /etc/ssh/sshd_host_dsa_* files have changed due to some reason.
Of course you should use this option only if you tend to access your remote host via a secured VPN or local network, otherwise the Host Key change could be an indicator someone is trying to intercept your ssh session.

 

Compression yes


– variable  enables compression of connection saves few bits was useful in the old modem telephone lines but still could save you few bits
It is also possible to define a full range of IP addresses to be accessed with one single public rsa / dsa key

Below .ssh/config
 

Host 192.168.5.?
     Hostname 192.168.2.18
     User admin
     IdentityFile ~/.ssh/id_ed25519.pub


Would instruct each host attemted to be reached in the IP range of 192.168.2.1-254 to be automatically reachable by default with ssh client with admin user and the respective ed25519.pub key.
 

$ ssh 192.168.1.[1-254] -v

 

2. Adding ssh client options system wide for all existing local or remote LDAP login users


The way to add any Host block is absolutely the same as with a default user except you need to add the configuration to /etc/ssh/ssh_config. Here is a confiugaration from mine Latest Debian Linux

$ cat /etc/ssh/ssh_config

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

As you can see pretty much can be enabled by default such as the forwarding of the Authentication agent option ( -A ) option, necessery for some Company server environments to be anbled. So if you have to connect to remote host with enabled Agent Forwarding instead of typing

ssh -A user@remotehostname


To enable Agent Forwarding instead of

ssh -X user@remotehostname


Simply uncomment and set to yes
 

ForwardX11 yes
ForwardX11Trusted yes


Just simply uncomment above's config ForwardAgent no

As you can see ssh could do pretty much, you can configure enable SSH Tunneling or run via a Proxy with the ProxyCommand (If it is the first time you hear about ProxyCommand I warmly recommend you check my previous article – How to pass SSH traffic through a secured Corporate Proxy Server with corkscrew).

Sometimes for a defines hostname, due to changes on remote server ssh configuration, SSH encryption type or a host key removal you might end up with issues connecting, therefore to override all the previously defined options inside .ssh/config by ignoring the configuration with -F /dev/null

$ ssh -F /dev/null user@freak -v


What we learned ?

To sum it up In this article, we have learned how to easify the stressed sysadmin life, by adding Aliases with certain port numbering and configurations for different remote SSH administrated Linux / Unix, hosts via local ~/.ssh/config or global wide /etc/ssh/ssh_config configuration options, as well as how already applied configuration from ~/.ssh/config affecting each user ssh command execution, could be overriden.

Remove “Windows 7 PC is out of Support” annoying reoccuring warning popup alert


September 10th, 2021

Windows-7-End-of-life-pc-is-out-of-support-removal-rip-win-7

Since January 15th 2020, Windows 7 which reached its End of Life (EOL)  and is no longer Supported. Windows 7 Service Pack 1 Starter, Home Basic, Home Premium, and Professional installations will display the message


"Your Windows 7 PC is out of support".

The use of Windows 7, since 2020 is steadily declining but some hard core maniacs, who refuse to be in tune with latest fashion do still roll Windows 7 on dedicated VPS Servers (running on Xen / VMWare etc.).
With the reach of End of Support, people who still run Windows 7 have no longer the usual Operating system provided.

  • No security updates
  • No software updates
  • No tech support

Even though running End of Support system is quite dangerous and you might get hacked easily by autometed bot, still for some custom uses and if the Windows 7 Runs behind a solid firewall it could be considered relatively safe.

Microsoft hence made their Windows (remote controlled system) to have an annoying pop up window with the "YOUR WINDOWS 7 PC IS OUT OF SUPPORT" as shown in below screenshot:

windows-7-disable-pc-is-out-of-support-popup-annoying-message-screenshot.

For those who don't plan to migrate from Windows 7 to Windows 10, this message becomes quickly very annoying especially if you happen to access remotely your Windows 7 VPS and use it for simple things as browsing a few news websites or you're a marketer and you use the Windows for accessing Amazon / Ebay from a different country as many Marketers do to access General Webstores emulating access from a remote location. 


Disable "Your Windows 7 PC is out of support" popup alert
 

Luckily it is possible to disable this annoying Your Windows 7 pop-up alert by setting a value key in Windows Registry
DiscontinueEOS to 1.

To do so launch from Administrator command  line cmd.exe prompt (or start it from Windows start menu):

regedit

1. Open Windows Registry Editor and navigate to

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\EOSNotify.

 You will need to set the DWORD DiscontinueEOS value to 1 in the Windows Registry
 

windows-7-disable-pc-is-out-of-support-popup-eosnotify-dword-registry-03-600x366

windows-7-disable-pc-is-out-of-support-popup-discontinueEOS-registry-modify

–  In case EOSNotify key is not available, right-click the CurrentVersion key and select New > Key and name it EOSNotify.

windows-7-disable-pc-is-out-of-support-popup-EOSNotify-create-new-key-600x367

2. Right click anywhere in the right pane and select New > DWORD (32-bit) Value and name it DiscontinueEOS.

3. Set Value data to 1 and click OK.

windows-7-disable-pc-is-out-of-support-popup-edit-dword-32-bit-value-regedit-screenshot.

4. When the new value has been set, Restart the Windows7 computer / Virtual machine, to make sure registry setting take effect.

windows-7-disable-pc-is-out-of-support-popup-discontninueEOS-reg-dword-0x000000001-600x248

 

To automate the procedure in large environments, you can create a small script using the reg  command load the Registry key or use Windows GPO (Group Policy Object) to enforce the setting across all Active Directory PC members.
 

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\EOSNotify]
"DiscontinueEOS"=dword:00000001
custom GPO in Active Directory.

To avoid potential issues using a non supported OS, you should start planning to upgrade your Windows 7 clients to Windows 10.

That's it ! Out of support Windows 7  alert should no longer bug you 🙂